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Abstract 

At the heart of every loop, and hence of all algorithms, lies a loop in- 
variant: a property ensured by the initialization and maintained by every 
iteration so that, when combined with the exit condition, it yields the 
loop's final effect. Identifying the invariant of every loop is not only a 
required step for software verification, but also a key requirement for un- 
derstanding the loop and the program to which it belongs. The systematic 
study of loop invariants of important algorithms can, as a consequence, 
yield insights into the nature of software. 

We performed this study over a wide range of fundamental algorithms 
from diverse areas of computer science. We analyze the patterns according 
to which invariants are derived from postconditions, propose a classifica- 
tion of invariants according to these patterns, and present its applica- 
tion to the algorithms reviewed. The discussion also shows the need for 
high-level specifications and invariants based on "domain theory". The 
included invariants and the corresponding algorithms have been mechani- 
cally verified using an automatic program prover. Along with the classifi- 
cation and applications, the conclusions include suggestions for automatic 
invariant inference and general techniques for model-based specification. 
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1 Introduction: inductive invariants 



At the core of any correctness verification for imperative programs lies the ver- 
ification of loops, for which the accepted technique (in the dominant approach, 
"axiomatic" or "Hoare-style" ) relies on associating with every loop a loop in- 
variant. Finding suitable loop invariants is a crucial and delicate step to veri- 
fication. Although some programmers may see invariant elicitation as a chore 
needed only for formal program verification, the concept is in fact widely useful, 
including for informal development: the invariant gives fundamental informa- 
tion about a loop, showing what it is trying to achieve and why it achieves it, 
to the point that (in some people's view at least) it is impossible to understand 
a loop without knowing its invariant. 

To explore and illustrate this view we have investigated a body of repre- 
sentative loop algorithms in several areas of computer science, to identify the 
corresponding invariants, and found that they follow a set of standard patterns. 
We set out to uncover, catalog, classify, and verify these patterns, and report 
our findings in the present article. 

Finding an invariant for a loop is traditionally the responsibility of a human: 
either the person performing the verification, or the programmer writing the 
loop in the first place (a better solution when applicable, using the constructive 
approach to programming advocated by Dijkstra and others [T5l [23l [39]). More 
recently, techniques have been developed for inferring invariants automatically, 
or semi-automatically with some human help (we review them in Section [SJ. We 
hope that the results reported here will be useful in both cases: for humans, to 
help obtain the loop invariants of new or existing programs, a task that many 
programmers still find challenging; and for invariant inference tools. 

For all algorithms presented in the papeiQ, we wrote fully annotated im- 
plementations and processed the result with the Boogie program verifier |36|. 
providing proofs of correctness. The Boogie implementations are available ato 

http : / /bitbucket . org/ sechairethz/ver if ied/ 

This verification result reinforces the confidence in the correctness of the algo- 
rithms presented in the paper and their practical applicability. 

The rest of this introductory section recalls the basic properties of invari- 
ants. Section [5] introduces a style of expressing invariants based on "domain 
theory" , which can often be useful for clarity and expressiveness. Section [3] 
presents two independent classifications of loop invariant clauses, according to 
their role and syntactic similarity with respect to the postcondition. Section 2] 
presents 19 algorithms from various domains; for each algorithm, it presents 
an implementation in pseudo-code annotated with complete specification and 
loop invariants. Section [5] discusses some related techniques to infer invariants 

1 With the exception of those in Sections l4.5l and l4.7l whose presentation is at a higher level 
of abstraction, so that a complete formalization would have required complex axiomatization 
of geometric and numerical properties beyond the focus of this paper. 

2 In the repository, the tag inv_survey identifies only the algorithms described in the paper; 
see |httpT7 /goo . gl/DsdrV for instruction on how to access it. 
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or other specification elements automatically. Section [5] draws lessons from the 
verification effort. Section [7] concludes. 

1.1 Loop invariants basics 

The loop invariants of the axiomatic approach go back to Floyd [18] and 
Hoare [26j . For this approach and for the present article, a loop invariant is 
not just a quantity that remains unchanged throughout executions of the loop 
body (a notion that has also been studied in the literature), but more specifically 
an "inductive invariant", of which the precise definition appears next. Program 
verification also uses other kinds of invariant, notably class invariants [271 140] . 
which the present discussion surveys only briefly in Section 11.41 

The notion of loop invariant is easy to express in the following loop syntax 
taken from Eiffel: 



1 


from 


2 


Init 


3 


invariant 


4 


Inv 


5 


until 


6 


Exit 


7 


variant 


8 


Var 


9 


loop 


10 


Body 


11 


end 



(the variant clause helps establish termination as discussed below). Init and 
Body are each a compound (a list of instructions to be executed in sequence); 
either or both can be empty, although Body normally will not. Exit and Inv 
(the inductive invariant) are both Boolean expressions, that is to say, predicates 
on the program state. The semantics of the loop is: 

1. Execute Init. 

2. Then, if Exit has value True, do nothing; if it has value False, execute 
Body, and repeat step [2] 

Another way of stating this informal specification is that the execution of the 
loop body consists of the execution of Init followed by zero or more executions 
of Body, stopping as soon as Exit becomes True. 

There are many variations of the loop construct in imperative programming 
languages: "while" forms which use a continuation condition rather than the 
inverse exit condition; "do-until" forms that always execute the loop body at 
least once, testing for the condition at the end rather than on entry; "for" or 
"do" forms ("across" in Eiffel) which iterate over an integer interval or a data 
structure. They can all be derived in a straightforward way from the above 
basic form, on which we will rely throughout this article. 



4 



The invariant Inv plays no direct role in the informal semantic specification, 
but serves to reason about the loop and its correctness. Inv is a correct invariant 
for the loop if it satisfies the following conditions: 

1. Any execution of Init, started in the state preceding the loop execution, 
will yield a state in which Inv holds. 

2. Any execution of Body, started in any state in which Inv holds and Exit 
does not hold, will yield a state in which Inv holds again. 

If these properties hold, then any terminating execution of the loop will yield 
a state in which both Inv and Exit hold. This result is a consequence of the loop 
semantics, which defines the loop execution as the execution of Init followed 
by zero or more executions of Body, each performed in a state where Exit does 
not hold. If Init ensures satisfaction of the invariant, and any one execution of 
Body preserves it (it is enough to obtain this property for executions started in 
a state not satisfying Exit), then Init followed by any number of executions of 
Body will. 

Formally, the following classic inference rule |27j uses the invariant to express 
the correctness requirement on any loop: 

{P} Init {Inv}, {Inv A not Exit} Body {Inv} 
{P} from Init until Exit loop Body end {Inv A Exit} 

This is a partial correctness rule, useful only for loops that terminate. Proofs 
of termination are in general handled separately through the introduction of 
a loop variant: a value from a well-founded set, usually taken to be the set 
of natural numbers, which decreases upon each iteration (again it is enough to 
show that it does so for initial states not satisfying Exit). Since in a well-founded 
set all decreasing sequences are finite, the existence of a variant expression 
implies termination. The rest of this discussion concentrates on the invariants; it 
only considers terminating algorithms, of course, and includes the corresponding 
variant clauses, but does not explain why the corresponding expression are 
indeed loop variants (non- negative and decreasing). 

If a loop is equipped with an invariant, proving its partial correctness means 
establishing the two hypotheses in the above rules: 

• {P} Init {Inv}, stating that the initialization ensures the invariant, is 
called the initiation property. 

• {Inv A not Exit} Body {Inv}, stating that the body preserves the invari- 
ant, is called the consecution (or inductiveness) property. 

1.2 A constructive view 

We may look at the notion of loop invariant from the constructive perspective 
of a programmer directing his program to reach a state satisfying a certain 
desired property, the postcondition. In this view program construction is a 
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form of problem-solving, and the various control structures are problem-solving 
techniques [T5} |39l |23l [41] ; a loop solves a problem through successive approxi- 
mation. 

Exit condition 




Figure 1: A loop as a computation by approximation. 



The idea of this solution, illustrated by Figure [TJ is the following: 

• Generalize the postcondition (the characterization of possible solutions) 
into a broader condition: the invariant. 

• As a result, the postcondition can be defined as the combination ("and" 
in logic, intersection in the figure) of the invariant and another condition: 
the exit condition. 

• Find a way to reach the invariant from the previous state of the compu- 
tation: the initialization. 

• Find a way, given a state that satisfies the invariant, to get another, still 
satisfying the invariant but closer, in some appropriate sense, to the exit 
condition: the body. 

For the solution to reach its goal after a finite number of steps we need a notion 
of discrete "distance" to the exit condition. This is the loop variant. 

The importance of the above presentation of the loop process is that it 
highlights the nature of the invariant: it is a generalized form of the desired 
postcondition, which in a special case (represented by the exit condition) will 
give us that postcondition. This view of the invariant as a particular way of 
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generalizing the desired goal of the loop computation explains why the loop in- 
variant is such an important property of loops; one can argue that understanding 
a loop means understanding its invariant (in spite of the obvious observation 
that many programmers write loops without ever formally learning the notion 
of invariant, although we may claim that if they understand what they are doing 
they are relying on some intuitive understanding of the invariant anyway, like 
Moliere's Mr. Jourdain speaking in prose without knowing it). 

The key results of this article can be described as generalization strategies 
to obtain invariants from postconditions. 

1.3 A basic example 

To illustrate the above ideas, the 2300-year-old example of Euclid's algorithm, 
while very simple, is still a model of elegance. The postcondition of the algorithm 
is 

Result = gcd(a, b) 

where the positive integers a and b are the input and gcd is the mathemati- 
cal Greatest Common Divisor function. The generalization is to replace this 
condition by 

Result = x A gcd(Result, x) — gcd(a, b) (1) 

with a new variable x, taking advantage of the mathematical property that, for 
every y, 

gcd(x, x) = x (2) 

The second conjunct, a generalization of the postcondition, will serve as the 
invariant; the first conjunct will serve as the exit condition. To obtain the loop 
body we take advantage of another mathematical property: for every x > y 

gcd(x,y)=gcd(x-y,y) (3) 

yielding the well-known algorithm in Figure [2j (As with any assertion, writing 
clauses successively in the invariant is equivalent to a logical conjunction.) This 
form of Euclid's algorithm uses subtraction; another form, given in Section l4.2.21 
uses integer division. 

We may use this example to illustrate some of the orthogonal categories in 
the classification developed in the rest of this article: 

• The last clause of the invariant is an essential invariant, representing a 
weakening of the postcondition. The first two clauses are a bounding 
invariant, indicating that the state remains within certain general bound- 
aries, and ensuring that the "essential" part is defined. 

• The essential invariant is a conservation invariant, indicating that a cer- 
tain quantity remains equal to its original value. 
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1 from 

2 Result := a ; x := b 

3 invariant 

4 Result >0 

5 x >0 

6 gcd (Result, x) — gcd (a, b) 

7 until 

8 Result = x 

9 loop 

10 if Result >x then 

11 Result := Result — x 

12 else Here x is strictly greater than Result 

13 x := x — Result 

14 end 

15 variant 

16 max (a, b) 

17 end 

Figure 2: Greatest common divisor with substraction. 

• The strategy that leads to this conservation invariant is uncoupling, which 
replaces a property of one variable (Result), used in the postcondition, 
by a property of two variables (Result and x), used in the invariant. 

The proof of correctness follows directly from the mathematical property 
stated: ([2]) establishes initiation, and ([3]) establishes consecution. 

Section [4.2.21 shows how the same techniques works is applicable backward, 
to guess likely loop invariant given the algorithm annotated with pre and post- 
condition, whose mutation yields a suitable loop invariant. 

1.4 Other kinds of invariant 

Loop invariants are the focus of this article, but before we return to them it 
is useful to list some other kinds of invariant encountered in software. (Yet 
other invariants, which lie even further beyond the scope of this discussion, play 
fundamental roles in fields such as physics; consider for example the invariance of 
the speed of light under a Lorentz transformation, and of time under a Galilean 
transformation.) 

In object-oriented programming, a class invariant (also directly supported 
by the Eiffel notation [TB]) expresses a property of a class that: 

• Every instance of the classes possess immediately after creation, and 

• Every exported feature (operation) of the class preserves, 

with the consequence that whenever such an object is accessible to the rest of the 
software it satisfies the invariant, since the life of an object consists of creation 
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followed by any number of "qualified" calls x.fto exported features / by clients 
of the class. The two properties listed are strikingly similar to initiation and 
consecution for loop invariants, and the connection appears clearly if we model 
the life of an object as a loop: 

1 from 

2 create x.make Written in some languages as x := new C() 

3 invariant 

4 CI The class invariant 

5 until 

6 "x is no longer needed" 

7 loop 

8 x. some_feature_of_the_class 

9 end 

Also useful are Lamport-style invariants [34| used to reason about concurrent 
programs, which obviate the need for the "ghost variables" of the Owicki-Gries 
method [43] ) . Like other invariants, a Lamport invariant is a predicate on the 
program state; the difference is that the definition of the states involves not only 
the values of the program's variables but also the current point of the execution 
of the program ("program counter" or PC) and, in the case of a concurrent 
program, the collection of the PCs of all its concurrent processes. An example 
of application is the answer to the following problem posed by Lamport [35] . 

Consider N processes numbered from through N — 1 in which each 
process i executes 
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B [i] := 1 
£\ : y[i] := x[(i - 1) mod JV] 

and stops, where each x[i] initially equals 0. (The reads and writes 
of each x[i] are assumed to be atomic.) [. . . ] The algorithm [. . . ] 
maintains an inductive invariant. Do you know what that invariant 
is? 

If we associate a proposition @(m,i) for m = 1,2,3 that holds precisely when 
the execution of process i reaches location l % ml an invariant for the algorithm 
can be expressed as: 

/ @(0, (i - 1) mod JV) A y[i] = \ 

V 

@(2,j)=> @(1, (i — 1) mod JV) A y[i] = 1 

V 

\ @(2, (i - 1) mod JV) A y[i] = 1 ) 

Yet another kind of invariant occurs in the study of dynamical systems, 
where an invariant is a region ICS of the state space S such that any trajectory 
starting in / or entering it stays in / indefinitely in the future: 

Vx £ I,Vt e T : $(t,z) 6 I 
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where T is the time domain and <£> : T x $ — > $ is the evolution function. The 
connection between dynamical system invariants and loop invariants is clear in 
the constructive view (Section II .2[) . and can be formally derived by modeling 
programs as dynamical systems or using some other operational formalism |20j . 
The differential invariants introduced in the study of hybrid systems |47) are 
also variants of the invariants defined by dynamical systems. 

2 Expressing invariants: domain theory 

To discuss and compare invariants we need to settle on the expressiveness of the 
underlying invariant language: what do we accept as a loop invariant? 

The question involves general assertions, not just invariants; more generally, 
we must make sure that any convention for invariants is compatible with the 
general scheme used for pre/post specification, since an invariant is a mutation 
(often a weakening) of the postcondition. 

The mathematical answer to the basic question is simple: an assertion other 
than a routine postcondition, in particular a loop invariant, is a predicate on 
the program state. For example the assertion x > 0, where x is a program 
variable, is the predicate that holds of all computation states in which the value 
of that variable is positive. (Another view would consider it as the subset of the 
state space containing all states that satisfy the condition; the two views are 
equivalent since the predicate is the characteristic function of the subset, and 
the subset is inverse domain of "true" for the predicate.) 

A routine postcondition is usually a predicate on two states, since the spec- 
ification of a routine generally relates new values to original ones. For example 
an increment routine yields a state in which the counter's value is one more 
on exit than on entry. The old notation, available for postconditions in Eiffel 
and other programming languages supporting contracts, reflects this need; for 
example a postcondition clause could read counter — old counter + 1. Other 
notations, notably the Z specification language [52], have a notation for "new" 
rather than "old" , as in counter' = counter + 1 where the primed variable de- 
notes the new value. Although invariants are directly related to postconditions, 
we will be able in this discussion to avoid such notations and treat invariants as 
one-state functions. (Technically this is always possible by recording the entry 
value as part of the state.) 

Programming languages offer a mechanism directly representing predicates 
on states: Boolean expressions. This construct can therefore be used - as in the 
x > example - to represent assertions; this is what assertion-aware program- 
ming languages typically do, often extending it with special notations such as 
old and support for quantifiers. 

This basic language decision leaves open the question of the level of expres- 
siveness of assertions. There are two possibilities: 

• Allow assertions, in particular postconditions and loop invariants, to use 
functions and predicates defined using some appropriate mechanism (of- 
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ten, the programming language's function declaration construct) to ex- 
press high-level properties based on a domain theory covering specifics of 
the application area. We call this approach domain theory. 

• Disallow this possibility, requiring assertions always to be expressed in 
terms of the constructs of the assertion language, without functions. We 
call this approach atomic assertions. 

The example of Euclid's algorithm above, simple as it is, was already an 
example of the domain-theory-based approach because of its use of a function 
gcd in the invariant clause 

gcd(Result,x) = gcd(a, b) (4) 
corresponding to a weakening of the routine postcondition 

Result = gcd(a, b) 

It is possible to do without such a function by going back to the basic 
definition of the greatest common denominator. In such an atomic-assertion 
style, the postcondition would read 

Result >0 (Alternatively, Result > 1) 

a \\ Result = (Result divides a) 

b \\ Result = (Result divides b) 

Vi € IN: (a \\i = 0)A (b \\i = 0) implies i< Result (Result is the greatest of all 

the numbers that satisfy the 
preceding properties) 

Expressing the invariant in the same style requires several more lines since the 
definition of the greatest common divisor must be expanded for both sides of ((4]). 

Even for such a simple example, the limitations of the atomic-assertion style 
are clear: because it requires going back to basic logical constructs every time, 
it does not scale. 

Another example where we can contrast the two styles is any program that 
computes the maximum of an array. In the atomic-assertion style, the postcon- 
dition will be written 

VfceZ: a.lower< k< a.upper implies a[k] < Result (Every element between 

bounds has a value smaller 
than Result) 

3k £ Z: a.lower< k< a.upper A a[k] = Result (Some element between 

bounds has a the value 
Result) 

This property is the definition of the maximum and hence needs to be writ- 
ten somewhere. If we define a function "max" to capture this definition, the 
specification becomes simply 

Result = max(a) 
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The difference between the two styles becomes critical when we come to 
the invariant of programs computing an array's maximum. Two different algo- 
rithms appear in Section B~T1 The first (Section 14.1. 1[) is the most straightfor- 
ward; it moves an index i from a. lower + 1 to a. upper, updating Result if the 
current value is higher than the current result (initialized to the first element 
a [a. lower]). With a domain theory on arrays, the function max will be avail- 
able as well as a notion of slice, where the slice a [i .. j] for integers i and j is 
the array consisting of elements of a in the range Then the invariant is 

simply 

Result = max(a [a. lower., i]) 

which is ensured by initialization and, on exit when i = a.upper, yields the 
postcondition Result = max(a) (based on the domain-theory property that 
a [a. lower .. a. upper] — a). The atomic- assertion invariant would be a variant 
of the expanded postcondition: 

Vfc G Z: a.lower< k<i implies a[k] < Result 
3k G Z: a.lower< k<i A a[k] = Result 

Consider now a different algorithm for the same problem (Section I4.1.2[) , 
which works by exploring the array from both ends, moving the left cursor i up 
if the element at i is less than the element at j and otherwise moving the right 
cursor j down. The atomic-assertion invariant adds a level of quantification: 

^ / Vfc G Z : a. lower < k < a. upper implies a[k] < m \ , , 
m ' ^ 3fc G Z : i < k < j and a[k] = m J ^ ' 

This gives an appearance of great complexity even though the invariant is con- 
ceptually very simple, capturing in a nutshell the essence of the algorithm (as 
noted earlier, one of the applications of a good invariant is that it enables us to 
understand the core idea behind a loop): 

max(a) = max(a[i..j]) 

In words: the maximum of the entire array is to be found in the slice that has 
not been explored yet. On exit, where i = j, we are left with a one-element 
slice, whose value (this is a small theorem of the corresponding domain theory) 
is its maximum and hence the maximum of the whole array. The domain-theory 
invariant makes the algorithm and its correctness immediately clear. In contrast, 
the atomic-assertion invariant ([S]) simply obscures the idea of the algorithm. 

The domain-theory approach means that before any attempt to reason about 
an algorithm we should develop an appropriate model of the underlying domain, 
by defining appropriate concepts such as greatest common divisor for algorithms 
on integers and slices and maximum for algorithms on arrays, establishing the 
relevant theorems (for example that x > y gcd(x, y) = gcd(a; — y, y) and 
that max(a[«..i]) = a[i]). These concepts and theorems need only be developed 
once for every application domain of interest, not anew for every program over 
that domain. The programs can then use the corresponding functions in their 
assertions, in particular the loop invariants. 
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The domain-theory approach takes advantage of standard abstraction mech- 
anism of mathematics. Its only practical disadvantage, for assertions embedded 
in a programming language, is that the functions over a domain (such as gcd) 
must come from some library and, if themselves written in the programming 
language, must satisfy strict limitations; in particular they must be "pure" 
functions defined without any reference to imperative constructs. This issue 
only matters, however, for the practical embedding of invariants in programs; 
it is not relevant to the conceptual discussion of invariants, independent of any 
implementation concerns, which is the focus of this paper. 

The remainder of this article relies, for each class of algorithms, on the 
appropriate domain theory, whose components (functions and theorems) are 
summarized at the beginning of the corresponding section. We will make no 
further attempt at going back to the atomic-assertion style; the examples above 
should suffice to show how much simplicity is gained through this policy. 

3 Classifying invariants 

Loop invariants and their constituent clauses can be classified along two dimen- 
sions: 

• By their role with respect to the postcondition (Section 13.11) , leading us 
to distinguish between "essential" and "bounding" invariant properties. 

• By the transformation technique that yields the invariant from the post- 
condition (Section 13.21) . Here we have techniques such as uncoupling and 
constant relaxation. 

3.1 Classification by role 

In the typical loop strategy described in Section 11.21 it is essential that succes- 
sive iterations of the loop body remain in the convergence regions where the 
generalized form of the postcondition is defined. The corresponding conditions 
make up the bounding invariant; the clauses describing the generalized post- 
condition is the essential invariant. The bounding invariant for the greatest 
common divisor algorithm consists of the clauses 

Result > 

x > 

The essential clause is 

gcd(Result, x) — gcd(a, b) 

yielding the postcondition if Result = x. 

For the one-way maximum program, the bounding invariant is 

a. lower <i< a. upper 
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and the essential invariant is 



Result = max (a [a. lower., i] ) 

yielding the postcondition when i = a.upper. Note that the essential invariant 
would not be defined without the bounding invariant, since the slice o [1.. i] 
would be undefined (if i > a.upper) or would be empty and have no maximum 
(if i < a. lower). 

For the two-way maximum program, the bounding invariant is 
a. lower < i < j < a.upper 
and the essential invariant is 

max(a) = max(a[i..j]) 

yielding the postcondition when i = j. Again the essential invariant would not 
be always defined without the bounding invariant. 

The separation between bounding and essential invariants is often straight- 
forward as in these examples. In case of doubt, the following observation will 
help distinguish. The functions involved in the invariant (and often, those of 
the postcondition) are often partial; for example: 

• gcd(ii, v) is only defined if u and v are both non-zero (and, since we con- 
sider natural integers only in the example, positive). 

• For an array a and an integer i , a[i] is only defined if i £[a. low er .. a.upper], 
and the slice a[i .. j] is non-empty only if [ i .. j] C[a.lower.. a.upper]. 

• max(a) is only defined if the array a is not empty. 

Since the essential clauses, obtained by postcondition generalization, use 
gcd(Result, x) and (in the array algorithms) array elements and maxima, the 
invariant must include the bounding clauses as well to ensure that these essen- 
tial clauses are meaningful. A similar pattern applies to most of the invariants 
studied below. 

3.2 Classification by generalization technique 

The essential invariant is a mutation (often, a weakening) of the loop's postcon- 
dition. The following mutation techniques are particularly common: 

Constant relaxation: replace a constant n (more generally, an expression 
which does not change during the execution of the algorithm) by a variable 
i, and use i — n as part or all of the exit condition. This is the technique 
used in the one-way array maximum computation, where the constant is 
the upper bound of the array. The invariant generalizes the postcondition 
"Result is the maximum of the array up to a. lower", where a. lower is a 
constant, with "Result is the maximum up to z" . This condition is trivial 
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to establish initially for a non-empty array (take i to be a. lower), easy to 
extend to an incremented i (take Result to be the greater of its previous 
value and a [i]), and yields the postcondition when i reaches a. upper. As 
we will see in Section |4.1.4[ binary search differs from sequential search by 
applying double constant relaxation, to both the lower and upper bounds 
of the array. 

Uncoupling: replace a variable v (often Result) by two (for example Result 
and x), using their equality as part or all of the exit condition. This 
strategy is used in the greatest common divisor algorithm. 

Term dropping: remove a subformula (typically a conjunct), which gives a 
straightforward weakening. This strategy is used in the partitioning algo- 
rithm (Section PTE]) . 

Aging: replace a variable (more generally, an expression) by an expression that 
represents the value the variable had at previous iterations of the loop. 
This typically accommodates "off-by-one" discrepancies between when a 
variable is evaluated in the invariant and when it is updated in the loop 
body. 

Backward reasoning: compute the loop's postcondition from another asser- 
tion by backward substitution. This can be useful for nested loops, where 
the inner loop's postcondition can be derived from the outer loop's invari- 
ant. 

4 The invariants of important algorithms 

The following subsections include a presentation of several algorithms, their loop 
invariants and their connection with each algorithm's postcondition. Table [1] 
lists the algorithms and their category. For more details about variants of 
the algorithms and their implementation, we refer to standard textbooks on 
algorithms [55 1 151 151 ] . 

4.1 Array searching 

Many algorithmic problems can be phrased as search over data structures - 
from the simple arrays up to graphs and other sophisticated representations. 
This section illustrates some of the basic algorithms operating on arrays. 

4.1.1 Maximum: one-variable loop 

The following routine max-onejway returns the maximum element of an un- 
sorted array a of bounds a. lower and a. upper. The maximum is only defined for 
a non-empty array, thus the precondition a. count > 1. The postcondition can 
be written 

Result = max(a) 



15 



Algorithm Type Section 

Maximum search (one variable) searching § 14.1.11 

Maximum search (two variable) searching § 14.1.21 

Sequential search in unsorted array searching § 14.1.31 

Binary search searching § 14.1.41 

Integer division arithmetic § 14.2.11 

Greatest common divisor (with division) arithmetic § 14.2.21 

Exponentiation (by squaring) arithmetic § 14.2.31 

Long integer addition arithmetic § 14.2.41 

Quick sort's partitioning sorting § 14.3.11 

Selection sort sorting § 14.3.21 

Insertion sort sorting § 14.3.31 

Bubble sort (basic) sorting § 14.3.41 

Bubble sort (improved) sorting § 14.3.51 

Comb sort sorting § 14.3.61 

Knapsack with integer weights dynamic programming § 14.4.11 

Levenstein distance dynamic programming § 14.4.21 

Rotating calipers algorithm computational geometry § 14.51 

List reversal data structures § 14.61 

PageRank algorithm fixpoint § 14.71 



Table 1: The algorithms presented in Section |4j 

Writing it in slice form, as Result =max(a [a.lower..a.upper]) yields the 
invariant by constant relaxation of either of the bounds. We choose the second 
one, a. upper, yielding the essential invariant clause 

Result = max(a [a. lower., i]) 

Figure [3] shows the resulting implementation of the algorithm. 

Proving initiation is trivial. Consecution relies on the domain-theory prop- 
erty that 

max(a [1.. = max(max(a [1.. i]),a [i + 1]) 

4.1.2 Maximum: two-variable loop 

The one-way maximum algorithm results from arbitrarily choosing to apply con- 
stant relaxation to either a. lower or (as in the above version) a. upper. Guided 
by a symmetry concern, we may choose double constant relaxation, yielding 
another maximum algorithm max-two-way which traverses the array from both 
ends. If i and j are the two relaxing variables, the loop body either increases i 
or decreases j. When i — j, the loop has processed all of a, and hence i and j 
indicate the maximum element. 

The specification (precondition and postcondition) is the same as for the 
previous algorithm. Figure U shows an implementation. 

It is again trivial to prove initiation. Consecution relies on the following two 
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l max_one_way (a: ARRAY [T]): T 



2 require 

3 a. count > 1 a. count is the number of elements of the array 

4 local 

5 i: INTEGER 

6 do 

7 from 

8 i := a. lower ; Result := a [a.lower] 

9 invariant 

10 a.lower < i < a. upper 

11 Result = max (a [a.lower, i]) 

12 until 

13 i = a.upper 

14 loop 

15 i := » + 1 

16 if Result <a [i] then Result := a [i] end 

17 variant 

18 a. length — i + 1 

19 end 

20 ensure 

21 Result = max (a) 

22 end 



Figure 3: Maximum: one-variable loop. 

domain-theory properties: 

j>i A a [i] >a [j] ==>• max(a [ i .. j]) = max(a [i.-j — 1]) (6) 
«<j A a [j] >a[i] max(a [ i .. j]) = max(a [« + (7) 

4.1.3 Search in an unsorted array 

The following routine has sequential returns the position of an occurrence of 
an element key in an array a or, if key does not appear, a special value. The 
algorithm applies to any sequential structure but is shown here for arrays. For 
simplicity, we assume that the lower bound a.lower of the array is 1, so that we 
can choose as the special value. Obviously this assumption is easy to remove 
for generality: just replace 0, as a possible value for Result, by a.lower — 1. 

The specification may use the domain-theory notation elements (a) to ex- 
press the set of elements of an array a. A simple form of the postcondition is 

Result t^O <^=> key € elements(a) (8) 
which just records whether the key has been found. We will instead use a form 
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l max.two.way (a: ARRAY [J]): T 



2 require 

3 a. count > 1 

4 local 

5 i, j: INTEGER 

6 do 

7 from 

8 i := a. lower ; j := a. upper 

9 invariant 

10 a. lower < i < j < a. upper 

11 max (a [i--j]) = max (a) 

12 until 

13 i — j 

14 loop 

15 if a [i] > a [j] then j := j — 1 else j := j + 1 end 

16 variant 

17 j i 

18 end 

19 Result := a [i] 

20 ensure 

21 Result = max (a) 

22 end 



Figure 4: Maximum: two-variable loop. 

that also records where the element appears if present: 

Result ^ =^> key = a [Result] (9) 
Result = => fcey ^ elements (a) (10) 

to which we can for clarity prepend the bounding clause 

Result <= [0.. a. upper] 

to make it explicit that the array access in (JSJ is defined when needed. 

If in dTU|) we replace a by a [1.. a. upper], we obtain the loop invariant of 
sequential search by constant relaxation: introducing a variable i to replace 
either of the bounds 1 and a. upper. Choosing the latter yields the following 
essential invariant: 

Result e [0, i] 

Result 7^ key = a [Result] 

Result = key $ elements (a [L. i]) 

leading to an algorithm that works on slices 1.. i for increasing i, starting at 
and with bounding invariant < i < a. count, as shown in Figure [5^1 

3 Note that in this example it is OK for the array to be empty, so there is no precondition on 
a. upper, although general properties of arrays imply that a. upper > 0; the value corresponds 
to an empty array. 
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1 has.sequential (a: ARRAY [T] ; key: Tj: INTEGER 



2 require 

3 a. lower =1 For convenience only, may be removed (see text). 

4 local 

5 i: INTEGER 

6 do 

7 from 

8 i := ; Result := 

9 invariant 

10 < i < a. count 

11 Result G [0, i] 

12 Result =>• /oej/ = a [Result] 

13 Result = => fej/ elements (a 

14 until 

15 i = a. upper 

16 loop 

17 i := i + 1 

18 if a [ i] = key then Result := i end 

19 variant 

20 a. upper — i + 1 

21 end 

22 ensure 

23 Result € [0, a.upper] 

24 Result / =>• /oej/ = a [Result] 

25 Result = key elements (a) 



26 end 

Figure 5: Search in an unsorted array. 

To avoid useless iterations the exit condition may be replaced by i = a.upper 
V Result >0. 

To prove initiation, we note that initially Result is and the slice a [1.. i] 
is empty. Consecution follows from the domain-theory property that, for all 
1 < i < a. upper. 

key G elements(a [1.. i+1]) key G elements(a [1.. i]) V key — a 

4.1.4 Binary search 

Binary search works on sorted arrays by iteratively halving a segment of the 
array where the searched element may occur. The search terminates either 
when the element is found or when the segment becomes empty, implying that 
the element appears nowhere in the array. 

As already remarked by Knuth many years ago [3TJ Vol. 3, Sec. 6.2.1] 

Although the basic idea of binary search is comparatively straightfor- 
ward, the details can be surprisingly tricky, and many programmers 
have done it wrong the first few times they tried. 
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Reasoning carefully on the specification (at the domain-theory level) and the 
resulting invariant helps avoid mistakes. 

For the present discussion it is interesting that the postcondition is the same 
as for sequential search (Section 14. 1 .3[) . so that we can see where the general- 
ization strategy differs, taking advantage of the extra property that the array is 
sorted. 

The algorithm and implementation now have the precondition 

sorted (a) 

where the domain-theory predicate sorted (a), defined as 

Vj G [a. lower .. a. upper — 1] : a [j] < a [j'+l] 

expresses that an array is sorted upwards. The domain theory theorem on 
which binary search rests is that for any value mid in [ i .. j] (where i and j are 
valid indexes of the array) and any value key of type T (the type of the array 
elements) 

(key < a[mid] A key G elements{a[i .. mid]) \ 
V 
key > a[mid] A key G elements(a[mid+l..j\) J 

(11) 

This property leads to the key insight behind binary search, whose invariant 
follows from the postcondition by variable introduction, mid serving as that 
variable. 

Formula is not symmetric with respect to i and j ; a symmetric version 
is possible, using in the second disjunct, ">" rather than ">" and mid rather 
than mid + 1. The form given in (jlip has the advantage of using two mutually 
exclusive conditions in the comparison of key to a [mid]. As a consequence, we 
can limit ourselves to a value mid chosen in [ i .. j— 1] (rather than [ i .. j] ) since 
the first disjunct does not involve j and the second disjunct cannot hold for 
mid = j (the slice a [mid + l..j] being then empty). All these observations and 
choices have direct consequences on the program text, but are better handled 
at the specification (theory) level. 

We will start for simplicity with the version ([8]) of the postcondition that 
only records presence or absence, repeated here for ease of reference: 

Result t^O <^=> key G elements(a) (12) 

Duplicating the right-hand side of (1121) . writing a in slice form a[l.. a. upper], 
and applying constant relaxation twice, to the lower bound 1 and the upper 
bound a. upper, yields the essential invariant: 

key G elements(a[i .. j]) key G elements(a) (13) 

with the bounding invariant 

1 < i < mid<j < a. upper 
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l has.binary (a: ARRAY [T] ; key: T): INTEGER 



2 require 

3 a. lower =1 For convenience, see comment about hassequential. 

4 a. count >0 

5 sorted (a) 

6 local 

7 i, j, mid: INTEGER 

8 do 

9 from 

10 i:= 1; j := a. upper; mid :=0; Result := 

11 invariant 

12 1 < i < mid<j < a. upper 

13 key £ elements (a[i..j]) key g elements (a) 

14 until 

15 i = j 

16 loop 

17 mid := "A value in — 1]" In practice chosen as i + (j — i)//2 

18 if a [mid] < key then i := mid +1 else j := mici end 

19 variant 

20 j — i 

21 end 

22 if a [mid] = key then Result := mid end 

23 ensure 

24 < Result < n 

25 Result / =>• A;ey = a [Result] 

26 Result = => fcey elements (a) 

27 end 



Figure 6: Binary search. 

which combines the assumptions on mid necessary to apply (fTTj) - also assumed 
in (|13|) - and the additional knowledge that 1 < i and j < a. upper. 
The attraction of this presentation is that: 

• The two clauses key < a[mid] and key > a[mid] of (1111) are easy-to-test 
complementary conditions, suggesting a loop body that preserves the in- 
variant by testing key against a [mid] and going left or right as a result 
of the test. 

• When i = j - the case that serves as exit condition - the left side of 
the equivalence (|13|) reduces to key — a [mid]; evaluating this expression 
tells us whether key appeared at all in the entire array, the information 
we seek. In addition, we can obtain the stronger postcondition, (|5j)- ([lTj)) , 
which gives Result its precise value, by simply assigning mid to Result. 

This leads to the implementation in Figure [6] 

To prove initiation, we note that initially Result is 0; so is mid, so that 
mid€ [i.j] is false. Consecution follows directly from (fTTj). 
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For the expression assigned to mid in the loop, given above in pseudocode 
as "A value in [i .. j— 1]", the implementation indeed chooses, for efficiency, the 
midpoint of the interval [ i .. j] , which may be written i + (j — i) // 2 where 
"//" denotes integer division. In an implementation, this form is to be preferred 
to the simpler (i + j) // 2, whose evaluation on a computer may produce an 
integer overflow even when i, j and their midpoint are all correctly representable 
on the computer's number system, but (because they are large) the sum i + j 
is not [3]. In such a case the evaluation of j — i is instead safe. 

4.2 Arithmetic algorithms 

Efficient implementations of the elementary arithmetic operations known since 
grade school require non-trivial algorithmic skills and feature interesting invari- 
ants, as the examples in this section demonstrate. 

4.2.1 Integer division 

The algorithm for integer division by successive differences inputs computes 
the integer quotient q and the remainder r of two integers m and n. The 
postcondition reads 

< r < m 
n = m ■ q + r 

The loop invariant consists of a bounding clause and an essential clause. The 
latter is simply an element of the postcondition: 

n = m ■ q + r 

The bounding clause weakens the other postcondition clause by keeping only its 
first part. 

< r 

so that the dropped condition r < m becomes the exit condition. As a conse- 
quence, r > m holds in the loop body, and the assignment r := r — m maintains 
the invariant property < r. It is straightforward to prove the implementation 
in Figure [5] correct with respect to this specification. 

4.2.2 Greatest common divisor (with division) 

Euclid's algorithm for the greatest common divisor offers another example where 
clearly separating between the underlying mathematical theory and the imple- 
mentation yields a concise and convincing correctness argument. Sections 11.31 
and [2] previewed this example by using the form that repeatedly subtracts one 
of the values from the other; here we will use the version that uses division. 

The greatest common divisor gcd(a, b) is the greatest integer that divides 
both a and 6, defined by the following axioms, where a and b are nonnega- 
tive integers such that at least one of them is positive ("\\" denotes integer 
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l divided.diff (n, m: INTEGER): (q, r. INTEGER) 



2 


require 




3 


n >0 




4 


m >0 




5 


do 




6 


from 




7 


r := n; 


q := 


8 


invariant 




9 


< r 




10 


n = m ■ 


q + r 


11 


until 




12 


r < m 




13 


loop 




14 


r := r - 


- m 


15 


9 := 9 - 


f 1 


16 


variant r 




17 


end 




18 


ensure 




19 


< r < m 




20 


n = m ■ q + r 


21 


end 





Figure 7: Integer division. 

remainder) : 

o\\ gcd(a, b) = 
6\\gcd(o,6) = 

Vd £ IN : (o\\d = 0) A (6\\d = 0) =^> d < gcd(a, 6) 

From this definition follow several properties of the gcd function: 

Commutativity: gcd(a, b) = gcd(6, a) 

Zero divisor: gcd(a, 0) = a 

Reduction: for & > 0, gcd(a, 6) = gcd(a\\6, a) 

The following property of the remainder operation is also useful: 

Nonnegativity: for integers a > and b > 0: a\\b > 

From the obvious postcondition Result =gcd(a, 6), we obtain the essential 
invariant in three steps: 

1. By backward reasoning, derive the loop's postcondition x = gcd(a, b) from 
the routine's postcondition Result =gcd(a,6). 

2. Using the zero divisor property, rewrite it as gcd(x, 0) = gcd(a, b). 
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l gcd_Euchd_division (a, b: INTEGER): INTEGER 



2 


require 


3 


a >0 


4 


b >0 


5 


local 


6 


f, x, INTEGER 


7 


do 


8 


from 


9 


x := a 


10 


y ~ b 


11 


invariant 


12 


x >0 


13 


2/ >0 


14 


ffcrf (x, y) = gcrf (a, b) 


15 


until 


16 


y = 


17 


loop 


18 


t := y 


19 


y := x \\ y 


20 


x :— t 


21 


variant y 


22 


end 


23 


Result := x 


24 


ensure 


25 


Result = gcd (a, b) 


20 


end 



Figure 8: Greatest common divisor with division. 

3. Apply constant relaxation, introducing variable x to replace 0. 

This gives the essential invariant gcd(a;, y) = gcd(a, b) together with the bound- 
ing invariants x > and y > 0. The corresponding implementation is shown in 
Figure H0 

Initiation is established trivially. Consecution follows from the reduction 
property. Note that unlike in the difference version (Section 1 1.3p . we can arbi- 
trarily decide always to divide x by y, rather than having to find out which of 
the two numbers is greater; hence the commutativity of gcd is not used in this 
proof. 

4.2.3 Exponentiation by successive squaring 

Suppose we do not have a built-in power operator and wish to compute m™. We 
may of course multiply m by itself n — 1 times, but a more efficient algorithm 

4 The variant is simply y, which is guaranteed to decrease at every iteration and be bounded 
from below by the property < x\\y < y. 
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squares to for all Is values in the binary representation of n. In practice, there 
is no need to compute this binary representation. 

l power .binary (m, n: INTEGER): INTEGER 



2 require 

3 n >0 

4 local 

5 x, y: INTEGER 

6 do 

7 from 

8 Result := 1 

9 x :— m 

10 y :— n 

11 invariant 

12 y > 

13 Result ■ x v = m n 

14 until y = 

15 loop 

16 if y. is_even then 

17 x :— x * x 

18 y:=y//2 

19 else 

20 Result := Result * x 

21 y := y - 1 

22 end 

23 variant y 

24 end 

25 ensure 

26 Result = m™ 

27 end 



Figure 9: Exponentiation by successive squaring. 
Given the postcondition 

Result = m n 

we first rewrite it into the obviously equivalent form Result • l 1 = to™. Then, 
the invariant is obtained by double constant relaxation: the essential property 



is easy to obtain initially (by setting Result, x, and y to 1, to and n), yields 
the postcondition when y — 0, and can be maintained while progressing towards 
this situation thanks to the domain-theory properties 



x 



2z _ (Jl\2zl2 



(x 2 ) 2z/2 (14) 
x-x*- 1 (15) 
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Using only (fl5l) would lead to the inefficient (n — l)-multiplication algorithm, 
but we may use (jT4j) for even values of y — 2z. This leads to the algorithm in 
Figure [51 

Proving initiation is trivial. Consecution is a direct application of the (|14l) 
and (|T5|) properties. 

4.2.4 Long integer addition 

The algorithm for long integer addition computes the sum of two integers a 
and b given in any base as arrays of positional digits starting from the least 
significant position. For example, the array sequence (3,2,0,1) represents the 
number 138 in base 5 as 3 ■ 5° + 2 • 5 1 + • 5 2 + 1 • 5 3 = 138. For simplicity 
of representation, in this algorithm we use arrays indexed by 0, so that we can 
readily express the value encoded in base b by an array a as the sum: 

a. count 

The postcondition of the long integer addition algorithm has two clauses. 
One specifies that the pairwise sum of elements in a and b encodes the same 
number as Result: 

n— 1 n 

^2(a[k] + b[k]) ■ base k = ^ Result [fc] • base k (16) 

k=Q k=0 

Result may have one more digit than a or 6; hence the different bound in the 
two sums, where n denotes a's and 6's length (normally written a. count and 
b. count). The second postcondition clause is the consistency constraint that 
Result is indeed a representation in base base: 

has-base (Result, base) (17) 

where the predicate has-base is defined by the following quantification over the 
array's length 

has-base (v, b) <^=> Vfc 6 IN : < k < v. count => < v[k] < b 

Both postcondition clauses appear mutated in the loop invariant. First, we 
rewrite Result in slice form Result [0..n] in (ITC1) and (fTT|) . The first essential 
invariant clause follows by applying constant relaxation to (|17[) , with the variable 
expression i — 1 replacing constant n: 

has-base (Result [0..i— 1], base) 

The decrement is required because the loops updates i at the end of each iter- 
ation; it is a form of aging (see Section [3.2[1 . 
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l addition (a, b: ARRAY [INTEGER] ; 



2 base: INTEGER): ARRAY [INTEGER] 

3 require 

4 base >0 

5 a. length = b. length = n > 1 

6 has-base (a, base) a is a valid encoding in base base 

7 has-base (b, base) 6 is a valid encoding in base base 

8 a. lower = b. lower — For simplicity of representation 

9 local 

10 i, d, carry: INTEGER 

11 do 

12 Initialize Result to an array of size n + 1 with all 0s 

13 Result := {0} n+1 

14 carry := 

15 from 

16 i := 

17 invariant 

18 Z)*=o( a [^ + b[k])-base k = carry-base 1 + ^."^Result^] -base k 

19 has_base (Result [0.. i— 1], base) 

20 < carry < base 

21 until 

22 i — n 

23 loop 

24 d :— a [i] + b [i] + carry 

25 Result [i] :— d \\ base 

26 carry := d // base 

27 i := i + 1 

28 variant 

29 n — i 

30 end 

31 Result [n] := carry 

32 ensure 

33 + 6[fc])-6ase fc = ELo Result W' 6asefc 

34 has_base (Result, aase) 

35 end 



Figure 10: Long integer addition. 

To get the other part of the essential invariant, we first highlight the last 
term in the summation on the right-hand side of (|1G|) : 

n—l Tl—l 

y^(a[k] + b[k]) ■ base k = Result [n] • base 1 + Result [fc] • base k 

k=0 k=0 

We then introduce variables i and carry, replacing constants n and Result [n]. 
Variable i is the loop counter, also mentioned in the other invariant clause; 
carry, as the name indicates, stores the remainder of each pairwise addition, 
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which will be carried over to the next digit. 

The domain property that the integer division by b of the sum of two 6-base 
digits vi , V2 is less than b (all variables are integer) : 

b > A v x ,v 2 6 [0..6- 1] => [vi +v 2 )//b G [0..6- 1] 

suggests the bounding invariant clause < carry <base. Figure ITOl shows the 
resulting implementation, where the most significant digit is set after the loop 
before terminating. 

Initiation is trivial under the convention that an empty sum evaluates to 
zero. Consecution easily follows from the domain-theoretic properties of the 
operations in the loop body, and in particular from how the carry and the 
current digit d are set in each iteration. 

4.3 Sorting 

A number of important algorithms sort an array based on pairwise comparisons 
and swaps of elements. The following domain-theory notations will be useful for 
arrays a and 6: 

• perm (a,b) expresses that the arrays are permutations of each other (their 
elements are the same, each occurring the same number of times as in the 
other array). 

• sorted (a) expresses that the array elements appear in increasing order: 
\fi <E[a.lower..a.upper — 1]: a [i] < a [i +1]. 

The sorting algorithms considered sort an array in place, with the specifica- 
tion 

sort (a: ARRAY [T]) 
require 

a. lower = 1 

a. length = n > 1 
ensure 

perm (a, old a) 

sorted (a) 

The type T indicates a generic type that is totally ordered and provides the 
comparison operators <, <, > 7 and >. The precondition that the array be 
indexed from 1 and non-empty is a simplification that can be easily dropped; 
we adopt it in this section as it focuses the presentation of the algorithms on 
the interesting cases. For brevity, we also use n as an alias of a's length a. count. 

The notation a[ i .. j] ~x, for an array slice a [ i .. j] , a scalar value x, and a 
comparison operator ~ among < , < , > , and > , denotes that all elements in the 
slice satisfy ~ with respect to x: it is a shorthand for Vfe £ a[k] ~ x. 



4.3.1 Quick sort: partitioning 



At the core of the well-known Quick sort algorithm lies the partitioning proce- 
dure, which includes loops with an interesting invariant; we analyze it in this 
section. 

l partition (a: ARRAY [T]; pivot: T): INTEGER 



2 require 

3 a. lower = 1 

4 a. length = n > 1 

5 local 

6 low, high: INTEGER 

7 do 

8 from low := 1 ; high := n 

9 invariant 

10 1 < low < n 

11 1 < high < n 

12 perm (a, old a) 

13 a [1.. low — 1] < pivot 

14 a [high + l..n] > pivot 

15 until low — high 

16 loop 

17 from This loop increases low 

18 invariant Same as outer loop 

19 until low = high V a[2o«j] > pivot 

20 loop low := low + 1 end 

21 

22 from This loop decreases high 

23 invariant Same as outer loop 

24 until low = high V a[high] < pivot 

25 loop high := high + 1 end 
26 

27 a. swap (low, high) Swap the elements in positions low and high 

28 variant high — low 

29 end 
30 

31 if a [low] > pivot then 

32 low := low — 1 

33 high := low 

34 end 

35 Result := low 

36 ensure 

37 < Result < n 

38 perm (a, old a) 

39 a [1.. Result] < pivot 

40 a [Result + l..n] > pivot 

41 end 



Figure 11: Quick sort: partitioning. 
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The procedure rearranges the elements in an array a according to an arbi- 
trary value pivot given as input: all elements in positions up to Result included 
are no larger than pivot, and all elements in the other "high" portion (after posi- 
tion Result) of the array are no smaller than pivot. Formally, the postcondition 
is: 

< Result < n 

perm (a, old a) 

a [1.. Result] < pivot 

a [Result + l..n] > pivot 

In the special case where all elements in a are greater than or equal to pivot, 
Result will be zero, corresponding to the "low" portion of the array being 
empty 

Quick sort works by partitioning an array, and then recursively partitioning 
each of the two portions of the partition. The choice of pivot at every recursive 
call is crucial to guarantee a good performance of Quick sort. Its correctness, 
however, relies solely on the correctness of partition , not on the choice of pivot. 
Hence the focus of this section on partition alone. 

The bulk of the loop invariant follows easily from the last three clauses of 
the postcondition, perm (a, old a) appears unchanged in the essential invariant, 
denoting the fact that the whole algorithm does not change a's elements but only 
rearranges them. The clauses comparing a's slices to pivot determine the rest 
of the essential invariant, once we modify them by introducing loop variables 
low and high decoupling and relaxing "constant" Result: 

perm (a, old a) 

a [1.. low — 1] < pivot 

a [high + l..n] > pivot 

The formula low = high - removed when decoupling - becomes the main loop's 
exit condition. Finally, a similar variable introduction applied twice to the 
postcondition < Result < n suggests the bounding invariant clauses 

1 < low < n 
1 < high < n 

The slice comparison a [1.. low — 1] < pivot also includes aging of variable 
low. This makes the invariant clauses fully symmetric, and suggests a matching 
implementation with two inner loops nested inside an overall outer loop. The 
outer loop starts with low = 1 and high = n and terminates, with low = high, 
when the whole array has been processed. The first inner loop increments low 
until it points to an element that is larger than pivot, and hence is in the wrong 
portion of the array. Symmetrically, the outer loop decrements high until it 
points to an element smaller than pivot. After low and high are set by the inner 
loops, the outer loop swaps the corresponding elements, thus making progress 



30 



towards partitioning the array. Figure [TT] shows the resulting implementation. 
The closing conditional in the main routine's body ensures that Result points 
to an element no greater than pivot; this is not enforced by the loop, whose 
invariant leaves the value of a [low] unconstrained. In particular, in the special 
case of all elements being no less than pivot, low and Result are set to zero 
after the loop. 

In the correctness proof, it is useful to discuss the cases a [ low] < pivot and 
a [ low] > pivot separately when proving consecution. In the former case, we 
combine a [1.. low — 1] < pivot and a [low] < pivot to establish the backward 
substitution a [1.. low] < pivot. In the latter case, we combine low = high, 
a [high + l..n] > pivot and a [low] > pivot to establish the backward substi- 
tution a [low., n] > pivot. The other details of the proof are straightforward. 

4.3.2 Selection sort 

Selection sort is a straightforward sorting algorithm based on a simple idea: to 
sort an array, find the smallest element, put it in the first position, and repeat 
recursively from the second position on. Pre- and postcondition are the usual 
ones for sorting (see Section [4. 3p . and hence require no further comment. 

The first postcondition clause perm (a, old a) is also an essential loop in- 
variant: 

perm (a, old a) (18) 

If we introduce a variable i to iterate over the array, another essential invariant 
clause is derived by writing a in slice form a [1.. n] and then by relaxing n into 
i: 

sorted (a [1.. i]) (19) 

with the bounding clause: 

1 < i < n (20) 

which ensures that the sorted slice a [1.. i] is always non-empty. The fi- 
nal component of the invariant is also an essential weakening of the post- 
condition, but is less straightforward to derive by syntactic mutation. If we 
split a [1.. n] into the concatenation a [1.. i — 1] o a [ i .. n], we notice that 
sorted (a [1.. i — 1] o a [i .. n]) implies 

Vfc G [i..n] : a [1.. i - 1] < a[k] (21) 

as a special case. (|2"Tj) guarantees that the slice a [i..n] that has not been 
sorted yet contains elements that are no smaller than any of those in the sorted 
slice a [1.. i — 1]. 

The loop invariants (TT8")) -(f2T) |) apply - possibly with minimal changes due 
to inessential details in the implementation - for any sorting algorithm that 
sorts an array sequentially, working its way from lower to upper indices. To 
implement the behavior specific to Selection sort, we introduce an inner loop 
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l selection_sort (a: ARRAY [T]) 



2 require 

3 a. lower = 1 

4 a. length = n > 1 

5 local 

6 i, J, m: INTEGER 

7 do 

8 from j := 1 

9 invariant 

10 1 < i < n 

11 perm (a, old a) 

12 sorted (a [1.. i]) 

13 Vfc G [i..n]: a - 1] < a [k] 

14 until 

15 i — n 

16 loop 

17 from j := i + 1 ; m := i 

18 invariant 

19 1< i <j < n + 1 

20 i < m <j 

21 perm (a, old a) 

22 sorted (a [1.. i]) 

23 a [1.. i — 1] < a [m] < a [ i .. j — 1] 

24 until 

25 j = n + 1 

26 loop 

27 if a [j] < a [m] then m := j end 

28 j := j + 1 

29 variant n — i — j 

30 end 

31 a. swap (i, m) Swap the elements in positions i and m 

32 i := i + 1 

33 variant n — i 

34 end 

35 ensure 

36 perm (a, old a) 

37 sorted (a) 

38 end 



Figure 12: Selection sort. 

that finds the minimum element in the slice a [ i .. n] , which is not yet sorted. To 
this end, it uses variables j and m: j scans the slice sequentially starting from 
position i + 1; m points to the minimum element found so far. Correspondingly, 
the inner loop's postcondition is a[m] < a[i..n], which induces the essential 
invariant clause 

a [m] < a - 1] (22) 
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specific to the inner loop, by constant relaxation and aging. The outer loop's 
invariant (|2Tj) clearly also applies to the inner loop - which does not change i 
or n - where it implies that the element in position m is an upper bound on all 
elements already sorted: 

a [1.. i - 1] < a [m] (23) 

Also specific to the inner loop are more complex bounding invariants relating 
the values of i, j, and m to the array bounds: 

l<i<j<n+l 
i < m < j 

The implementation in Figure [12] follows these invariants. The outer loop's only 
task is then to swap the "minimum" element pointed to by m with the lowest 
available position pointed to by i. 

The most interesting aspect of the correctness proof is proving consecution 
of the outer loop's invariant clause (fl"9|). and in particular that a [ i] < a[ i + 1]. 
To this end, notice that (|22|) guarantees that a [m] is the minimum of all el- 
ements in positions from i to n; and (|23|) that it is a upper bound on the 
other elements in positions from 1 to i — 1. In particular, a [m]< a and 
a [i — 1] < a[m] hold before the swap on line |3"T1 After the swap, a[i] equals 
the previous value of a[m], thus a[i — 1] < a [i] < a[i +1] holds as required. 
A similar reasoning proves the inductiveness of the other main loop's invariant 
clause (|2T]> . 

4.3.3 Insertion sort 

Insertion sort is another sub-optimal sorting algorithm that is, however, simple 
to present and implement, and reasonably efficient on arrays of small size. As 
the name suggests, insertion sort hinges on the idea of re-arranging elements in 
an array by inserting them in their correct positions with respect to the sorting 
order; insertion is done by shifting the elements to make room for insertion. 
Pre- and postcondition are the usual ones for sorting (see Section 14.31 and the 
comments in the previous subsections). 

The main loop's essential invariant is as in Selection sort (Section l4.3.2l) and 
other similar algorithms, as it merely expresses the property that the sorting 
has progressed up to position i and has not changed the array content: 

sorted {a [1.. i]) (24) 
perm (a, old a) (25) 

This essential invariant goes together with the bounding clause 1 < i < n. 

The main loop includes an inner loop, whose invariant captures the specific 
strategy of Insertion sort. The outer loop's invariant (|25[) must be weakened, 
because the inner loop overwrites a [i] while progressively shifting to the right 
elements in the slice a [1.. j]. If a local variable v stores the value of a [i] 
before entering the inner loop, we can weaken (|25|) as: 

perm (a ° v o a[j + 2..n], old a) (26) 
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l insertion_sort (A: ARRAY [T]) 



2 require 

3 a. lower = 1 ; a. length = n > 1 

4 local 

5 i, j: INTEGER ; v : T 

6 do 

7 from i := 1 

8 invariant 

9 1 < i < n 

10 perm (a, old a) 

n sorted (a [1.. i]) 

12 until i = n 

13 loop 

14 i := t + 1 

15 u := o [i] 

16 from j := i — 1 

17 invariant 

is < j < i < n 

19 perm (a [1.. j] o v o a[j + 2..n], old a) 

20 sorted (a [1.. 8 — 1]) 

21 sorted (a [j + 1.. i]) 

22 d < a [j + 1.. i] 

23 until j ' = or a [j] < v 

24 loop 

25 a [j + 1] := a [j] 

26 j := j - 1 

27 variant j ' — i 

28 end 

29 a [j + 1] := v 

30 variant n — i 

31 end 

32 ensure 

33 perm (a, old a) 

34 sorted (a) 

35 end 



Figure 13: Insertion sort. 

where "o" is the concatenation operator; that is, a's element at position j + 1 is 
the current candidate for inserting v — the value temporarily removed. After the 
inner loop terminates, the outer loop will put v back into the array in position 
j + 1 (line [55] in Figure H"3"f , thus restoring the stronger invariant (f^5|) (and 
establishing inductiveness for it). 

The clause (|24|) . crucial for the correctness argument, is also weakened in 
the inner loop. First, we "age" i by replacing it with i — 1; this corresponds 
to the fact that the outer loop increments i at the beginning, and will then 
re-establish (IM1) only at the end of each iteration. Therefore, the inner loop can 
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only assume the weaker invariant: 

sorted (a [1.. i - 1]) (27) 

that is not invalidated by shifting (which only temporarily duplicates elements). 
Shifting has, however, another effect: since the slice a[j + contains ele- 
ments shifted up from the sorted portion, the slice a[j + is itself sorted, 
thus the essential invariant: 

sorted (a [j + (28) 

We can derive the pair of invariants (|27)) . ([28)) from the inner loop's post- 
condition (|2"4")l : write a [1.. i] as a [1.. i — 1] o a[i .. i]; weaken the formula 
sorted (a [1.. i — 1] o a[i.. i]) into the conjunction of sorted( a [1.. i — 1]) 
and sorted (a[i .. j]); replace one occurrence of constant i in the second con- 
junct by a fresh variable j and age to derive sorted (a [j + 1.. i]) . 

Finally, there is another essential invariant, specific to the inner loop. Since 
the loop's goal is to find a position, pointed to by j + 1, before i where v can 
be inserted, its postcondition is: 

v< a [j + (29) 

which is also a suitable loop invariant, combined with a bounding clause that 
constrains j and i: 

< j <i<n (30) 

Overall, clauses (f26l) -([30 |) are the inner loop invariant; and Figure [T3l shows the 
matching implementation. 

As usual for this kind of algorithms, the crux of the correctness argument 
is proving that the outer loop's essential invariant is inductive, based on the 
inner loop's. The formal proof uses the following informal argument. (|27[) and 
(|2"5|) imply that inserting v at j + 1 does not break the sortedness of the slice 
a [1.. j + 1]. Furthermore, (|28| guarantees that the elements in the "upper" 
slice a [j + are also sorted with a [j] < a[j + 1] < a[j + 2]. (The de- 

tailed argument would discuss the cases j = 0, < j < i — 1, and j = i — 1.) In 
all, the whole slice a [1.. i] is sorted, as required by (f24|) . 

4.3.4 Bubble sort (basic) 

As a sorting method, Bubble sort is known not for its performance but for its 
simplicity [23 Vol. 3, Sec. 5.2.2]. Bubble sort relies on the notion of inversion: 
a pair of elements that are not ordered, that is such that the first is greater 
than the second. The straightforward observation that an array is sorted if and 
only if it has no inversions suggests to sort an array by iteratively removing 
all inversions. Let us present invariants that match such a high-level strategy, 
deriving them from the postcondition (which is the same as the other sorting 
algorithms of this section). 
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l bubble_sort_basic (a: ARRAY [T]) 



2 require 

3 a. lower = 1 ; a. length = n > 1 

4 local 

5 swapped: BOOLEAN 

6 i: INTEGER 

7 do 

8 from swapped := True 

9 invariant 

10 perm (a, old a) 

11 -i swapped =^sorted (a) 

12 until -i swapped 

13 loop 

14 swapped := False 

15 from i := 1 

16 invariant 

17 1 < i < n 

18 perm (a, old a) 

19 not swapped ==>sorted (a [1.. i]) 

20 until i — n 

21 loop 

22 if a [ i] >a [ i + 1] then 

23 a. swap («, ? + 1) Swap the elements in positions i and i + 1 

24 swapped := True 

25 end 

26 t := i + 1 

27 variant n — i 

28 end 

29 variant | inversions ( a) 

30 end 

31 ensure 

32 perm (a, old a) 

33 sorted (a) 

34 end 



Figure 14: Bubble sort (basic version). 

The postcondition perm (a, old a) that a's elements be not changed is also 
an invariant of the two nested loops used in Bubble sort. The other postcondi- 
tion sorted ( a) is instead weakened, but in a way different than in other sorting 
algorithms seen before. We introduce a Boolean flag swapped, which records if 
there is some inversion that has been removed by swapping a pair of elements. 
When swapped is false after a complete scan of the array a, no inversions have 
been found, and hence a is sorted. Therefore, we use -i swapped as exit condition 
of the main loop, and the weakened postcondition 

-i swapped => sorted (a) (31) 
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as its essential loop invariant. 

The inner loop performs a scan of the input array that compares all pairs 
of adjacent elements and swaps them when they are inverted. Since the scan 
proceeds linearly from the first element to the last one, we get an essential 
invariant for the inner loop by replacing n by i in pip written in slice form: 

-i swapped =^sorted (a [1.. i]) (32) 

The usual bounding invariant 1 < i < n and the outer loop's invariant clause 
perm (a, old a) complete the inner loop invariant. 

The implementation is now straightforward to write as in Figure 1141 The 
inner loop, in particular, sets swapped to True whenever it finds some inversion 
while scanning. This signals that more scans are needed before the array is 
certainly sorted. 

Verifying the correctness of the annotated program in Figure [TJ] is easy, 
because the essential loop invariants (|3"Tj) and (1521 are trivially true in all itera- 
tions where swapped is set to true. On the other hand, this style of specification 
makes the termination argument more involved: the outer loop's variant (line 1291 
in Figure I14[) must explicitly refer to the number of inversions left in a, which 
are decreased by complete executions of the inner loop. 

4.3.5 Bubble sort (improved) 

The inner loop in the basic version of Bubble sort - presented in Section 14.3.41 
- always performs a complete scan of the n-element array a. This is often 
redundant, because swapping adjacent inverted elements guarantees that the 
largest misplaced element is sorted after each iteration. Namely, the largest 
element reaches the rightmost position after the first iteration, the second-largest 
one reaches the penultimate position after the second iteration, and so on. This 
section describes an implementation of Bubble sort that takes advantage of this 
observation to improve the running time. 

The improved version still uses two nested loops. The outer loop's essential 
invariant: 

sorted (a [i + l..n]) (33) 

is a weakening of the postcondition that encodes the knowledge that the "upper" 
part of array a is sorted. Variable i is now used in the outer loop to mark 
the portion still to be sorted; correspondingly, the bounding invariant clause 
1 < i < n is also part of the outer loop's specification. 

Continuing with the same logic, the inner loop's postcondition: 

a [1.. i + 1] < a[i + 1] (34) 

states that the largest element in the slice a [1.. i + 1] has been moved to the 
highest position. Constant relaxation, replacing i (not changed by the inner 
loop) with a fresh variable j, yields a new essential component of the inner 
loop's invariant: 

a [1.. j] < a[j] (35) 
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l bubble_sort_improved (a: ARRAY [T]) 



2 require 

3 a. lower = 1 ; a. length = n > 1 

4 local 

5 i, j: INTEGER 

6 do 

7 from i :— n 

8 invariant 

9 1 < t < n 

10 perm (a, old a) 

11 sorted (a [i + l..n]) 

12 until i = 1 

13 loop 

14 from j := 1 

15 invariant 

16 1 < i < n 

17 1 < j < % + 1 

18 perm (a, old a) 

19 sorted (a [i + l..n]) 

20 a [1.. j] < a[j] 

21 until j = i + 1 

22 loop 

23 if a [j] > a [j + 1] then a.swap (j, j + 1) end 

24 j := j + 1 

25 variant i — j 

26 end 

27 i :— i — 1 

28 variant i 

29 end 

30 ensure 

31 perm (a, old a) 

32 sorted (a) 

33 end 



Figure 15: Bubble sort (improved version). 

whose aging (using j instead of j + 1) accommodates incrementing j at the 
end of the loop body. The outer loop's invariant and the bounding clause 
1 < j < i + 1 complete the specification of the inner loop. Figure [15] displays 
the corresponding implementation. 

The correctness proof follows standard strategies. In particular, the inner 
loop's postcondition (l34l) - i.e., the inner loop's invariant when j = i + 1 
implies a [i] < a[i +1] as a special case. This fact combines with the other 
clause ([33)) to establish the inductiveness of the main loop's essential clause: 

sorted (a[i .. n]) 

Finally, proving termination is trivial for this program because each loop has 
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an associated iteration variable that is unconditionally incremented or decre- 
mented. 



4.3.6 Comb sort 

In an attempt to improve performance in critical cases, Comb sort generalizes 
Bubble sort based on the observation that small elements initially stored in 
the right-most portion of an array require a large number of iterations to be 
sorted. This happens because Bubble sort swaps adjacent elements; hence it 
takes n scans of an array of size n just to bring the smallest element from the 
right-most nth position to the first one, where it belongs. Comb sort adds the 
flexibility of swapping non- adjacent elements, thus allowing a faster movement 
of small elements from right to left. A sequence of non-adjacent equally-spaced 
elements also conveys the image of a comb's teeth, hence the name "Comb sort" . 

Let us make this intuition rigorous and generalize the loop invariants, and the 
implementation, of the basic Bubble sort algorithm described in Section 14.3.41 
Comb sort is also based on swapping elements, therefore the - now well-known 
- invariant perm (o, old a) also applies to its two nested loops. To adapt the 
other loop invariant (|31j) . we need a generalization of the predicate sorted that 
fits the behavior of Comb sort. Predicate gapsorted (a, d), defined as: 

gapsorted{a,d) Vfc G [a. lower .. a. upper — d] : a [k] < a [k + d] 

holds for arrays a such that the subsequence of d-spaced elements is sorted. 
Notice that for d = 1: 



gapsorted reduces to sorted; this fact will be used to prove the postcondition 
from the loop invariant upon termination. 

With this new piece of domain theory, we can easily generalize the essential 
and bounding invariants of Figure [14] to Comb sort. The outer loop considers 
decreasing gaps; if variable gap stores the current value, the bounding invariant 



defines its variability range. Precisely, the main loop starts with with gap = n 
and terminates with gap = 1, satisfying the essential invariant: 



The correctness of Comb sort does not depend on how gap is decreased, as long 
as it eventually reaches 1; if gap is initialized to 1, Comb sort behaves exactly as 
Bubble sort. In practice, it is customary to divide gap by some chosen parameter 
c at every iteration of the main loop. 

Let us now consider the inner loop, which compares and swaps the subse- 
quence of d-spaced elements. The Bubble sort invariant (|32|) generalizes to: 




1 < gap < n 




(36) 



-i swapped =^gapsorted (o [1.. i — 1 + gap], gap) 



(37) 
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l comb.sort (a: ARRAY [T\) 



2 require 

3 a. lower = 1 ; a. length = n > 1 

4 local 

5 swapped: BOOLEAN 

6 i, gap: INTEGER 

7 do 

8 from swapped := True ; gap := n 

9 invariant 

10 1 < gap < n 

11 perm (a, old a) 

12 -i swapper =>• gapsorted (a, gap) 

13 until 

14 -i swapped and gap = 1 

15 loop 

16 gap := max (1, gap // c) 

17 c>lisa parameter whose value does not affect correctness 

18 swapped := False 

19 from i := 1 

20 invariant 

21 1 < gap < n 

22 1 < i < i + gap < n + 1 

23 perm (a, old a) 

24 ' swapped gapsorted (a — 1 + gap], gap) 

25 until 

26 i + gap = n+l 

27 loop 

28 if a [i] > a[i + gap] then 

29 a. swap (i, i + gap) 

30 swapped := True 

31 end 

32 i := i + 1 

33 variant n + 1 — gap — i end 

34 variant | inversions ( a) | end 

35 ensure 

36 perm (a, old a) 

37 sorted (a) 

38 end 



Figure 16: Comb sort. 

and its matching bounding invariant is: 

1 < i < i + gap < n + 1 

so that when i = n + 1 + gap the inner loop terminates and ()37j) is equivalent to 
(|36p . This invariant follows from constant relaxation and aging; the substituted 
expression i — 1 + gap is more involved to accommodate how i is used and 
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updated in the inner loop, but is otherwise semantically straightforward. 

The complete implementation is shown in Figure [TBI The correctness argu- 
ment is exactly as for Bubble sort in Section T4.3.41 but exploits the properties 
of the generalized predicate gapsorted instead of the simpler sorted. 

4.4 Dynamic programming 

Dynamic programming is an algorithmic technique used to compute functions 
that have a natural recursive definition. Dynamic programming algorithms 
construct solutions iteratively and store the intermediate results, so that the so- 
lution to larger instances can reuse the previously computed solution for smaller 
instances. This section presents a few examples of problems that lend themselves 
to dynamic programming solutions. 

4.4.1 Unbounded knapsack problem with integer weights 

We have an unlimited collection of items of n different types. An item of type 
k, for k = 1, . . . , n, has weight w[k] and value v[k]. The unbounded knapsack 
problem asks what is the maximum overall value that one can carry in a knap- 
sack whose weight limit is a given weight. The attribute "unbounded" refers to 
the fact that we can pick as many object of any type as we want: the only limit 
is given by the input value of weight, and by the constraint that we cannot store 
fractions of an item - either we pick it or we don't. 

Any vector s of n nonnegative integers defines a selection of items, whose 
overall weight is given by the scalar product: 

s ■ w = s[fc]u>[/c] 

l<k<n 

and whose overall value is similarly given by the scalar product s ■ v. Using 
this notation, we introduce the domain-theoretical function maxJtnapsack which 
defines the solution of the knapsack problem given a weight limit b and items 
of n types with weight and value given by the vectors w and v. 

(3s e IN" : s ■ w < b A s ■ v = n \ 
A 
VteW 1 : t-w <b =S> t-v < k ) 

that is, the largest value achievable with the given limit. Whenever weights 
w, values v, and number n of item types are clear from the context, we will 
abbreviate max-knapsack {b, v, w, n) by just K(b). 

The unbounded knapsack problem is NP-complete [HJ[3D]. It is, however, 
weakly NP-complete [44] , and in fact it has a nice solution with pseudo-polyno- 
mial complexity based on a recurrence relation, which suggests a straightforward 
dynamic programming algorithm. The recurrence relation defines the value of 
K{b) based on the values K (&') for b 1 < b. 

The base case is for b = 0. If we assume, without loss of generality, that no 
item has null weight, it is clear that we cannot store anything in the knapsack 
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without adding some weight, and hence the maximum value attainable with a 
weight limit zero is also zero: K(0) = 0. Let now & be a generic weight limit 
greater than zero. To determine the value of K(b), we make a series of attempts 
as follows. First, we select some item of type k, such that w[k] < b. Then, we 
recursively consider the best selection of items for a weight limit of b — w [k] ; 
and we set up a new selection by adding one item of type k to it. The new 
configuration has weight no greater than b — w [k] + w [k] = b and value 

v[k]+K(b-w[k]) 

which is, by inductive hypothesis, the largest achievable by adding an element of 
type k . Correspondingly, the recurrence relation defines K (b) as the maximum 
among all values achievable by adding an object of some type: 

6 = 

K{b) = { r ril , WL r;n . M , Jf _ r;1 i _ (38) 




+ K(b - w[k]) | k S [l..n] and < w[k] < &} 



b > 



The dynamic programming solution to the knapsack problem presented in 
this section computes the recursive definition (|38|) for increasing values of b. 
It inputs arrays v and w (storing the values and weights of all elements), the 
number n of element types, and the weight bound weight. The precondition 
requires that weight be nonnegative, that all element weights w be positive, and 
that the arrays v and w be indexed from 1 to n: 

weight > 
w > 

v. lower — w. lower — 1 
v. upper — w. upper = n 

The last two clauses are merely for notational convenience and could be dropped. 
The postcondition states that the routine returns the value K(weight) or, with 
more precise notation: 

Result = maxJtnapsack (weight, v, w, n) 

The main loop starts with 6 = and continues with unit increments until 
b = weight; each iteration stores the value of K(b) in the local array m, so that 
m [weight] will contain the final result upon termination. Correspondingly, the 
main loop's essential invariant follows by constant relaxation: 

• Variable m [weight] replaces constant Result, thus connecting m to the 
returned result. 

• The range of variables [1.. b] replaces constant weight, thus expressing the 
loop's incremental progress. 
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l knapsack (v, w: ARRAY [INTEGER]; n, weight: INTEGER): INTEGER 



2 require 

3 weight > 

4 w >0 

5 v. lower = w.lower = 1 

6 v. upper — w. upper = n 

7 local 

8 b, j: INTEGER 

9 m: ARRAY [INTEGER] 

10 do 

11 from b := ; m [0] := 

12 invariant 

13 < b < weight 

14 m [0..6] = maxjknapsack ([0..6], v, w, n) 

15 until b = weight 

16 loop 

17 6 := 6 + 1 

18 from j := ; m [b] := m [b — 1] 

19 invariant 

20 < fe < weight 

21 < j < n 

22 m [0.. 6 — 1] = maxjknapsack ([0..b — 1], v, w, n) 

23 m [6] = bestjualue (b, v, w, j, n) 

24 until j — n 

25 loop 

26 j := j + 1 

27 if «) [j] < b and m [b] <v [j] + m [b — w then 

28 m [b] := v [j] + m [b — w [j]] 

29 end 

30 variant n — j end 

31 variant weight — b end 

32 Result := m [weight] 

33 ensure 

34 Result = maxjknapsack (weight, v, w, n) 

35 end 



Figure 17: Unbounded knapsack problem with integer weights. 

The loop invariant it thus: 

Vy 6 [0..6] : m{y] = max_knapsack{y,v,w,n) (39) 
which goes together with the bounding clause 

< b < weight 

that qualifies 6's variability domain. With a slight abuse of notation, we con- 
cisely write (|39[) . and similar expressions, as: 

77i[0..&] = max-knapsack([0..b], v, w, n) (40) 
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The inner loop computes the maximum of definition psp iteratively, for 
all element types j, where 1 < j < n. To derive its essential invariant, we 
first consider its postcondition (similarly as the analysis of Selection sort in 
Section 14.3.21) . Since the inner loop terminates at the end of the outer loop's 
body, the inner's postcondition is the outer's invariant (14TJ|) . Let us rewrite it 
by highlighting the value m[b] computed in the latest iteration: 

m[0..b — 1] = maxJtnapsack ([0..6 — 1], v, w, n) (41) 
m[b] = best-value (b,v,w,n,n) (42) 

Function bestjvalue is part of the domain theory for knapsack, and it expresses 
the "best" value that can be achieved given a weight limit of b, j < n element 
types, and assuming that the values K (b') for lower weight limits b' < b are 
known: 

bestjvalue (b, v,w,j,n) — 

max ju[fc] + K(b - w[k}) | k £ and < w[k] < b\ 

If we substitute variable j for constant n in (|4"2"j) , expressing the fact that the 
inner loop tries one element type at a time, we get the inner loop essential 
invariant: 

m[0..b— 1] = maxJtnapsack ([0..6 — 1], v, w, n) 
m[b] = bestjvalue (b,v,w, j,m) 

The obvious bounding invariants < b < weight and < j < n complete the 
inner loop's specification. Figure [17] shows the corresponding implementation. 

The correctness proof reverses the construction we highlighted following the 
usual patterns seen in this section. In particular, notice that: 

• When j = n the inner loop terminates, thus establishing (14"T1) and (|4"2")l . 

• (|4T|) and (|42|) imply (|40|) because the recursive definition (|38|) for some b 
only depends on the previous values for b' < b, and (j4Tj) guarantees that 
m stores those values. 



4.4.2 Levenshtein distance 



The Levenshtein distance of two sequences s and t is the minimum number of 
elementary edit operations (deletion, addition, or substitution of one element in 
either sequence) necessary to turn s into t. The distance has a natural recursive 
definition: 



distance(s, t)=< 







n 

distance ( s[l..m - 1], t[l..n - 1] \ 

( distanee(s[l..m — l],t), \ 

i + min distance(s,t[l..n — 1]), 

i distance(s[l..m — l],*[l..n — 1]) J 



m = n = 

m > 0, n = 

n > 0, m = 

m > 0, n > 0, s[m] = t[n] 

m > 0, n > 0, s[m] ^ t[n] 
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where m and n respectively denote s's and i's length (written s. count and 
t. count when s and t are arrays). The first three cases of the definition are 
trivial and correspond to when s, t, or both are empty: the only way to get 
a non-empty string from an empty one is by adding all the former's elements. 
If s and t are both non-empty and their last elements coincide, then the same 
number of operations as for the shorter sequences s[l..m — 1] and t[l..n — 1] 
(which omit the last elements) are needed. Finally, if s's and i's last elements 
differ, there are three options: 1) delete s[m] and then edit s[l..m — 1] into t; 2) 
edit s into t[l..n — 1] and then add t[n] to the result; 3) substitute t[n] for s[m] 
and then edit the rest s[l..m — 1] into t[l..n — 1]. Whichever of the options 1), 
2), and 3) leads to the minimal number of edit operations is the Levenshtein 
distance. 

It is natural to use a dynamic programming algorithm to compute the Lev- 
enshtein distance according to its recursive definition. The overall specification, 
implementation, and the corresponding proofs, are along the same lines as the 
knapsack problem of Section 14.4.11 therefore, we briefly present only the most 
important details. The postcondition is simply 

Result = distance (s, t) 

The implementation incrementally builds a bidimensional matrix d of dis- 
tances such that the element d[i,j] stores the Levenshtein distance of the se- 
quences s[l..i] and t[l..j]. Correspondingly, there are two nested loops: the 
outer loop iterates over rows of d, and the inner loop iterates over each col- 
umn of d. Their essential invariants express, through quantification, the partial 
progress achieved after each iteration: 

Vhe [0.i-l],Vfce [0..n] :d[h,k] = distance(s[l..h],t[l..k]) 
Mh e [0..i - l],Vfc G [0..j - 1] : d[h,k] = distance(s[l..h],t[l..k]) 

The standard bounding invariants on the loop variables i and j complete the 
specification. 

Figure [TBI shows the implementation, which uses the compact across nota- 
tion for loops, similar to for loops in other languages. The syntax 

across [a., b] invariant I as k loop B end 

is simply a shorthand for: 

from k := a invariant / until k = b + 1 loop B ; k := k 
+ 1 end 

For brevity, Figure [18] omits the obvious loop invariants of the initialization 
loops at lines fTOl and ITTl 

4.5 Computational geometry: Rotating calipers 

The diameter of a polygon is its maximum width, that is the maximum distance 
between any pair of its points. For a convex polygon, it is clear that a pair of 
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l Levenshtein.distance (s, t: ARRAY [T\): INTEGER 



2 require 

3 s . count = m 

4 t . count — n 

5 local 

6 i, j: INTEGER 

7 d: ARRAY [INTEGER, INTEGER] 

8 do 

9 d:= {0} m+1 x {0} n+1 

10 across [l..m] as i loop d [i ,0] := i end 

11 across [l..n] as j loop d [0,j] := J end 

12 

13 across [1.. m] as i 

14 invariant 

15 1 < i < m + 1 

16 Vft G [0..i - l],Vfc G [0..m]: [ft, k] = distance (s [l..ft], i 

17 loop 

18 across [1.. n] as j 

19 invariant 

20 1 < j < m + 1 

21 1 < j < n + 1 

22 Vft, G [0..i - 1], VJfe G [0..j - 1]: d [h, k] = distance (s [l..h], t [l..k]) 

23 loop 

24 if s [i] = t [j] then 

25 d := d [i - 1, j - 1] 

26 else 

27 d := 1 + min (d [i -1,3- 1], d [i, j — 1], tf [i - 1, j]) 

28 end 

29 end 

30 end 

31 Result := d [m, n] 

32 ensure 

33 Result = distance (s, t) 

34 end 



Figure 18: Levenshtein distance. 

vertices determine the diameter. Shamos showed |51) that it is not necessary to 
check all 0(n 2 ) pairs of points: his algorithm, shown below, runs in time 0(n). 
The correctness of the algorithm rests on the notions of lines of support and 
antipodal points. A line of support is analogue to a tangent: a line of support 
of a convex polygon p is a line that intersects p and such that the interior of 
p lies entirely to one side of the line. An antipodal pair is then any pair of p's 
vertices that lie on two parallel lines of supports. It is a geometric property that 
an antipodal pair determines the diameter of any polygon p, and that a convex 
polygon with n vertices has 0(n) antipodal pairs. 

Shamos's algorithms efficiently enumerates all antipodal pairs by rotating 
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l diameter.calipers (p: LIST [POINT]): INTEGER 



2 require 

3 p. count > 3 ; p. is_convex 

4 local 

5 jaw_a, jaw-b: VECTOR Jaws of the caliper 

6 n_a, nj>: NATURAL Pointers to vertices of the polygon 

7 angle_a, anglej>: REAL Angle measures 

8 do 

9 n_a := "Index, in p, of the vertex with the minimum y coordinate" 
10 njj :— "Index, in p, of the vertex with the maximum y coordinate" 
ll 

12 jaw_a := "Horizontal direction from p[n_a] pointing towards negative" 

13 jawj) := "Horizontal direction from p[n_6] pointing towards positive" 

14 from 

15 totaljrotation := 

16 Result := |p[n_a] — p[ra_6]| Distance between pair of vertices 

17 invariant 

18 parallel (jaw_a, jawJb) Jaws are parallel 

19 < Result < diameter (p) Result increases until diameter (p) 

20 < totaljrotation < 360 

21 until totaljrotation > 180 All antipodal pairs considered 

22 loop 

23 angle_a := "Angle between p[n_a] and the next vertex in p " 

24 angleJ) := "Angle between p[n_6] and the next vertex in p " 

25 if angle_a < angleJ) then 

26 Rotate jau>-a to coincide with the edge p[n_a] — p[n_a].next 

27 jaw_a :— jaw_a + angle_a 

28 Rotate jaui-b by the same amount 

29 jaui-b := jawj) + angle_a 

30 Next current point n_a 

31 7i_a := "Index of vertex following p[n_a]" 

32 Update total rotation 

33 totaljrotation := totaljrotation + angle_a 

34 else 

35 As in the then branch with a's and 6's roles reversed 

36 end 

37 Update maximum distance between antipodal points 

38 Result := max (|p[n_a] — p[n_6]|, Result) 

39 variant 180 — totaljrotation end 

40 ensure 

41 Result = diameter (p) 

42 end 



Figure 19: Diameter of a polygon with rotating calipers. 



47 



two lines of support while maintaining them parallel. After a complete rotation 
around the polygon, they have touched all antipodal pairs, and hence the al- 
gorithm can terminate. Observing that two parallel support lines resemble the 
two jaws of a caliper, Toussaint [51] suggested the name "rotating calipers" to 
describe Shamos's technique. 

A presentation of the rotating calipers algorithm and a proof of its correct- 
ness with the same level of detail as the algorithms in the previous sections 
would require the development of a complex domain theory for geometric en- 
tities, and of the corresponding implementation primitives. Such a level of 
detail is beyond the scope of this paper; instead, we outline the essential traits 
of the specification and give a description of the algorithm in pseudo-code in 
Figure [T9E 

The algorithm inputs a list p of at least three points such that it represents 
a convex polygon (precondition on line [3]) and returns the value of the polygon's 
diameter (postcondition on line l4Tj) . 

It starts by adjusting the two parallel support lines on the two vertices with 
the maximum difference in y coordinate. Then, it enters a loop that computes 
the angle between each line and the next vertex on the polygon, and rotates 
both jaws by the minimum of such angles. At each iteration, it compares the 
distance between the new antipodal pair and the currently stored maximum 
(in Result), and updates the latter if necessary. Such an informal descrip- 
tion suggests the obvious bounding invariants that the two jaws are maintained 
parallel fline [T8|) . Result varies between some initial value and the final value 
diameter (p) (line |T9)) , and the total rotation of the calipers is between zero and 
180 + 180 (line [20)) . The essential invariant is, however, harder to state formally, 
because it involves a subset of the antipodal pairs reached by the calipers. A 
semi-formal presentation is: 



whose intended meaning is that Result stores the maximum distance between 
all points pi,P2 among p's vertices that can be reached with a rotation of up to 
totaLrotation degrees from the initial calipers' horizontal positions. 

4.6 Algorithms on data structures: List reversal 

Consider a list of elements of generic type G implemented as a linked list: each 
element's attribute next stores a reference to the next element in the list; and 
the last elements's next attribute is Void. This section discusses the classic 
algorithm that reverses a linked list iteratively. 

5 The algorithm in Figure [19] is slightly simplified, as it does not deal explicitly with the 
special case where a line initially coincides with an edge: then, the minimum angle is zero, 
and hence the next vertex not on the line should be considered. This problem can be avoided 
by adjusting the initialization to avoid that a line coincides with an edge. 
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We introduce a specification that abstracts some implementation details by 
means of a suitable domain theory. If list is a variable of type LIST [G] - that 
is, a reference to the first element - we lift the semantics of list in assertions 
to denote the sequence of elements found by following the chain of references 
until Void. This interpretation defines finite sequences only if list 's reference 
sequence is acyclic, which we write acyclic ( list). Thus, the precondition of 
routine reverse is simply 

acyclic ( list ) 

where list is the input linked list. 

For a sequence s = Si S2 ■ ■ ■ s n of elements of length n > 0, its reversal 
rev (s) is inductively defined as: 

rev(s) = \ (43) 
I rev(s2 ■ ■ ■ Sn) si n > 

where e denotes the empty sequence and "o" is the concatenation operator. 
With this notation, reverse's postcondition is: 

list = rev (old list ) 

with the matching property that list is still acyclic. 

The domain theory for lists makes it possible to derive the loop invariant 
with the usual techniques. Let us introduce a local variable reversed, which will 
store the iteratively constructed reversed list. More precisely, every iteration 
of the loop removes one element from the beginning of list and moves it to 
the beginning of reversed, until all elements have been moved. When the loop 
terminates: 

• list points to an empty list; 

• reversed points to the reversal of old list . 

Therefore, the routine concludes by assigning reversed to overwrite list . Back- 
ward substitution yields the loop's postcondition from the routine's: 

reversed = rev (old list ) (44) 

Using (|43[) for empty lists, we can equivalently write (|44l) as: 

rev( list ) o reversed = rev (old list ) (45) 

which is the essential loop invariant, whereas list — Void is the exit condi- 
tion. The other component of the loop invariant is the constraint that list and 
reversed be acyclic, also by mutation of the postcondition. 

Figure [2D] shows the standard implementation. Figure [5T] pictures instead 



a graphical representation of reverse's behavior: Figure 21(a) shows the state 



of the lists in the middle of the computation, and Figure 21(b) shows how the 



state changes after one iteration, in a way that the invariant is preserved. 
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1 reverse ( list : LIST [G\) 



2 
3 
4 
5 
6 
7 
8 
9 
10 
11 
12 
13 
14 
15 
16 
17 
18 
19 



20 ensure 

21 
22 



require 

acyclic ( list ) 
local 

reversed, temp: LIST [G\ 
do 

from reversed := Void 
invariant 

rev ( list ) o reversed = rei> (old foi ) 

acyclic ( iisi ) 

acyclic (reversed) 
until list — Void 
loop 

temp := list . next 

list .next := reversed 

reversed := list 

list := temp 
variant list . count end 
list := reversed 



list = rev (old list] 
acyclic ( list ) 



23 end 



Figure 20: Reversal of a linked list. 
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(a) Before executing the loop body. 

reversed temp list 
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(b) After executing the loop body: new links are in red. 

Figure 21: One iteration of the loop in routine reverse. 
The correctness proof relies on some straightforward properties of the rev 



!)0 



and o functions. Initiation follows from the property that soe = s. Consecution 
relies on the definition (|43|) for n > 0, so that: 

rev ( list . next) o list . first = rev ( list ) 

Proving the preservation of acyclicity relies on the two properties: 

acyclic(si S2 ■■■ s n ) =>■ acyclic(s2 ■■■ s n ) 
\si\ = 1 A acyclic(r) =>■ acyclic(si or) 

4.7 Fixpoint algorithms: PageRank 

PageRank is a measure of the popularity of nodes in a network, used by the 
Google Internet search engine. The basic idea is that the PageRank score of a 
node is higher the more nodes link to it (multiple links from the same page or 
self links are ignored). More precisely, the PageRank score is the probability 
that a random visit on the graph (with uniform probability on the outgoing 
links) reaches it at some point in time. The score also takes into account a 
dampening factor, which limits the number of consecutive links followed in a 
random visit. If the graph is modeled by an adjacency matrix (modified to 
take into account multiple and self links, and to make sure that there are no 
sink nodes without outgoing edges) , the PageRank scores are the entries of the 
dominant eigenvector of the matrix. 

In our presentation, the algorithm does not deal directly with the adjacency 
matrix but inputs information about the graph through arguments reaching and 
outbound. The former is an array of sets of nodes: reaching [k] denotes the set 
of nodes that directly link to node k. The other argument outbound [k] denotes 
instead the number of outbound links (to different nodes) originating in node 
k. The Result is a vector of n real numbers, encoded as an array, where n is 
the number of nodes. If eigenvector denotes the dominant eigenvector (also of 
length n) of the adjacency matrix (defined implicitly), the postcondition states 
that the algorithm computes the dominant eigenvector to within precision e 
(another input): 

\eigenvector — Result\ < e (46) 

That is, Result [k] is the rank of node k to within overall precision e. 

The algorithm computes the PageRank score iteratively: it starts assuming 
a uniform probability on the n nodes, and then it updates it until convergence 
to a fixpoint. Before every iteration, the algorithm saves the previous values of 
Result as old_rank, so that it can evaluate the progress made after the iteration 
by comparing the sum diff of all pairwise absolute differences, one for each node, 
between the scores saved in oldjrank and the newly computed scores available in 
Result. Correspondingly, the main loop's essential invariant is postcondition 
(|4"6"|) with diff substituted for e. diff gets smaller with every iteration of the 
main loop, until it becomes less than e, the main loop terminates, and the 
postcondition holds. The connection between the main loop invariants and the 
postcondition is thus straightforward. 
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l page-rank (dampening, e: REAL, reaching: ARRAY [SET [INTEGER]], 



2 outbound: ARRAY [INTEGER]): ARRAY [REAL] 

3 require 

4 < dampening < 1 

5 e >0 

6 reaching . count — outbound, count — n >0 

7 local 

8 diff: REAL 

9 old-rank: ARRAY [REAL] 

10 lmkJ,o: SET [INTEGER] 

11 do 

12 oldjrank := {l/n} n Initialized with n elements all equal to 1/n 

13 from diff := 1 

14 invariant 

15 | eigenvector — Result | < diff 

16 until diff < e 

17 loop 

is diff := 

19 across [l..n] as i loop 

20 Result [i] ■— 

21 link-to := reaching [i] 

22 across [1.. link-to . count] as j loop 

23 Result [i] := Result [i] + old-rank [j] / outbound [j] 

24 end 

25 Result [i] := dampening * Result [i] + (1 — dampening) /n 

26 diff := diff + |Result [i] — old-rank [i]\ 

27 end 

28 old-rank := Result Copy values of Result into old-rank 

29 variant 1 + diff — e 

30 end 

31 ensure 

32 | eigenvector — Result | <e 

33 end 



Figure 22: PageRank fixpoint algorithm. 
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Figure [22] shows an implementation of this algorithm. The two across loops 
nested within the main loop update the PageRank scores in Result. Every 
iteration of the outer across loop updates the value of Result [i] for node i as: 

(1 — dampening) . sr~^ old-rank[j] 

'■ h dampening ■ > — (47) 

n " / — J outbound[j\ 

The inner loop computes the sum in (|47[) for j ranging over the set reaching [ i] 
of nodes that directly reach i. The invariants of the across loops express the 
progress in the computation of (f4"71) : we do not write them down explicitly as 
they are not particularly insightful from the perspective of connecting postcon- 
ditions and loop invariants. 



5 Related work: Automatic invariant inference 

The amount of research work on the automated inference of invariants is sub- 
stantial and spread over more than three decades; this reflects the cardinal role 
that invariants play in the formal analysis and verification of programs. This sec- 
tion outlines a few fundamental approaches that emerged, without any pretense 
of being exhaustive. A natural classification of the methods to infer invariants is 
between static and dynamic. Static methods (Section [5T} use only the program 
text, whereas dynamic methods (Section l5.2[) summarize the properties of many 
program executions with different inputs. 



Program construction. In the introductory sections, we already mentioned 
classical formal methods for program construction [231 [Ml E] on which this 
survey paper is based. In particular, the connection between a loop's postcon- 
dition and its invariant buttresses the classic methods of program construction; 
this survey paper has demonstrated it on a variety of examples. In previous 
work [21] . we developed gin-pink, a tool that practically exploits the connection 
between postconditions and invariants. Given a program annotated with post- 
conditions, gin-pink systematically generates mutations based on the heuristics 
of Section 13.21 and then uses the Boogie program verifier [36] to check which 
mutations are correct invariants. The gin-pink approach borrows ideas from 
both static and dynamic methods for invariant inference: it is only based on 
the program text (and specification) as the former, but it generates "candidate" 
invariants to be checked like dynamic methods do. 



5.1 Static methods 

Historically, the earliest methods for invariant inference where static as in the 
pioneering work of Karr [29 . Abstract interpretation and the constraint-based 
approach are the two most widespread frameworks for static invariant inference 
(see also [51 Chap. 12]). 



53 



Abstract interpretation is a symbolic execution of programs over abstract 
domains that over-approximates the semantics of loop iteration. Since the sem- 
inal work by Cousot and Cousot [15], the technique has been updated and ex- 
tended to deal with features of modern programming languages such as object- 
orientation and heap memory- management (e.g., |37[ [7]). One of the main suc- 
cesses of abstract interpretation has been the development of sound but incom- 
plete tools [2] that can verify the absence of simple and common programming 
errors such as division by zero or void dereferencing. 

Constraint-based techniques rely on sophisticated decision procedures over 
non-trivial mathematical domains (such as polynomials or convex polyhedra) 
to represent concisely the semantics of loops with respect to certain template 
properties. 

Static methods are sound and often complete with respect to the class of 
invariants that they can infer. Soundness and completeness are achieved by 
leveraging the decidability of the underlying mathematical domains they rep- 
resent; this implies that the extension of these techniques to new classes of 
properties is often limited by undecidability. State-of-the-art static techniques 
can infer invariants in the form of mathematical domains such as linear inequal- 
ities [TTJ [5] , polynomials [551 ES] > restricted properties of arrays [5j HI [25] , and 
linear arithmetic with uninterpreted functions [T]. 

Following Section 13.11 the loop invariants that static techniques can easily 
infer are often a form of "bounding" invariant. This suggests that the special- 
ized static techniques for loop invariant inference discussed in this section, and 
the idea of deriving the loop invariant from the postcondition, demonstrated in 
the rest of the paper, can be fruitfully combined: the former can easily pro- 
vide bounding loop invariants, whereas the latter can suggest the "essential" 
components that directly connect to the postcondition. 

To our knowledge, there are only a few approaches to static invariant infer- 
ence that take advantage of existing annotations j45j [28j HH [33l [32] . Janota [28] 
relies on user-provided assertions nested within loop bodies and tries to check 
whether they hold as invariants of the loop. The approach has been evalu- 
ated only on a limited number of straightforward examples. De Caso et al. [2] 
briefly discuss deriving the invariant of a for loop from its postcondition, within 
a framework for reasoning about programs written in a specialized programming 
language. Lahiri et al. [33] also leverage specifications to derive intermediate 
assertions, but focusing on lower-level and type-like properties of pointers. On 
the other hand, Pasareanu and Visser [45] derive candidate invariants from 
postconditions within a framework for symbolic execution and model-checking. 

Finally, Kovacs and Voronkov [32] derive complex loop invariants by first en- 
coding the loop semantics as recurring relations and then instructing a rewrite- 
based theorem prover to try to remove the dependency on the iterator variables 
in the relations. This approach exploits heuristics that, while do not guarantee 
completeness, are practically effective to derive automatically loop invariants 
with complex quantification - a scenario that is beyond the capabilities of most 
other methods. 
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5.2 Dynamic methods 

Only in the last decade have dynamic techniques been applied to invariant infer- 
ence. The Daikon approach of Ernst et al. [T7] showed that dynamic inference 
is practical and sprung much derivative work (e.g., [35J [TU [3H [55J E2 an d many 
others). In a nutshell, the dynamic approach consists in testing a large number 
of candidate properties against several program runs; the properties that are 
not violated in any of the runs are retained as "likely" invariants. This implies 
that the inference is not sound but only an "educated guess" : dynamic invariant 
inference is to static inference what testing is to program proofs. Nonetheless, 
just like testing is quite effective and useful in practice, dynamic invariant infer- 
ence can work well if properly implemented. With the latest improvements [56], 
dynamic invariant inference can attain soundness of over 99% for the "guessed" 
invariants. 

Among the numerous attempts to improve the effectiveness and flexibility of 
dynamic inference, Gupta and Heidepriem [23] suggest to improve the quality 
of inferred contracts by using different test suites (i.e., based on code coverage 
and invariant coverage), and by retaining only the contracts that are inferred 
with both techniques. Fraser and Zeller (19j simplify and improve test cases 
based on mining recurring usage patterns in code bases; the simplified tests are 
easier to understand and focus on common usage. Other approaches to improve 
the quality of inferred contracts combine static and dynamic techniques (e.g., 

[Ml EH [56]). 

To date, dynamic invariant inference has been mostly used to infer pre and 
postconditions or intermediate assertions, whereas it has been only rarely ap- 
plied [42] to loop invariant inference. This is probably because dynamic tech- 
niques require a sufficiently varied collection of test cases, and this is more 
difficult to achieve for loops - especially if they manipulate complex data struc- 
tures. 

6 Lessons from the mechanical proofs 

Our verified Boogie implementations of the algorithms mirror the presentation 
in Section [31 as they introduce the predicates, functions, and properties of the 
domain theory that are directly used in the specification and in the correctness 
proofs. The technology of automatic theorem proving is, however, still in active 
development, and faces in any case insurmountable theoretical limits of unde- 
cidability. As a consequence, in a few cases we had to introduce explicitly some 
intermediate domain properties that were, in principle, logical consequences of 
other definitions, but that the prover could not derive without suggestion. Sim- 
ilarly, in other cases we had to guide the proof by writing down explicitly some 
of the intermediate steps in a form acceptable to the verifier. 

A lesson of this effort is that the syntactic form in which definitions and 
properties of the domain theory are expressed may influence the amount of 
additional annotations required for proof automation. The case of the sorting 
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algorithms in Section 14.31 is instructive. All rely on the definition of predicate 
sorted as: 

Vi G [a. lower .. a. upper — 1]: a [i] < a [i + 1] (48) 

which compares adjacent elements in positions i and i + 1. Since Bubble sort 
rearranges elements in an array also by swapping adjacent elements, proving 
it in Boogie was straightforward, since the prover could figure out how to ap- 
ply definition (|48[) to discharge the various verification conditions - which also 
feature comparisons of adjacent elements. Selection sort and Insertion sort re- 
quired substantially more guidance in the form of additional domain theory 
properties and detailing of intermediate verification steps, since their logic does 
not operate directly on adjacent elements, and hence their verification condi- 
tions are syntactically dissimilar to (|48|). Comb sort was as easy as Bubble sort, 
because it relics on a generalization of sorted that directly matches its logic (see 
Section |4~32]). 

A similar experience occurred for the two dynamic programming algorithms 
of Section l4~4l While the implementation of Levenshtein distance closely mirrors 
the recursive definition of distance ( Section 14.4. 2[) in computing the minimum, 
the relation between specification and implementation is less straightforward 
for the knapsack problem (Section I4.4.1|) , which correspondingly required more 
complicated axiomatization and proof in Boogie. 

7 Conclusions and assessment 

The concept of loop invariant is, as this review has attempted to show, one of 
the foundational ideas of software construction. We have seen many examples of 
the richness and diversity of practical loop invariants, and how they illuminate 
important algorithms from many different areas of computer science. We hope 
that these examples establish the assertion, made at the start of the article, 
that the invariant is the key to every loop: to devise a new loop so that it is 
correct requires summoning the proper invariant; to understand an existing loop 
requires understanding its invariant. 

Invariants belong to a number of categories, for which this discussion has 
established a classification which we hope readers will find widely applicable, 
and useful in understanding the loop invariants of algorithms in their own fields. 
The classification is surprisingly simple; perhaps other researchers will find new 
criteria that have eluded us. 

Descriptions of algorithms in articles and textbooks has increasingly, in re- 
cent years, included loop invariants, at least informally; we hope that the present 
discussion will help reinforce this trend and increase the awareness - not only in 
the research community but also among software practitioners - of the centrality 
of loop invariants in programming. 

Informal invariants are good, but being able to express them formally is even 
better. Reaching this goal and, more generally, continuing to make invariants 
ever more mainstream in software development requires convenient, clear and 
expressive notations for expressing loop invariants and other program assertions. 
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One of the contributions of this article will be, we hope, to establish the prac- 
ticality of domain-theory-based invariants, which express properties at a high 
level of abstraction, and their superiority over approaches that always revert to 
the lowest level. 

Methodological injunctions are necessary, but the history of software practice 
shows that they only succeed when supported by effective tools. Many program- 
mers still find it hard to come up with invariants, and this survey shows that 
they have some justifications: even though the basic idea is often clear, coming 
up with a sound and complete invariant is an arduous task. Progress in invari- 
ant inference, both theoretical and on the engineering side, remains essential. 
There is already, as noted, considerable work on this topic, often aimed at infer- 
ring more general invariant properties than the inductive loop invariants of the 
present discussion; but much remains to be done to bring the tools to a level 
where they can be integrated in a standard development environment and rou- 
tinely suggest invariants, conveniently and correctly, whenever a programmer 
writes a loop. The work mentioned in Section [5] is a step in this direction. 

Another major lessons for us from preparing this survey (and reflecting on 
how different it is from what could have been written on the same topic 30, 
20, f or even 5 years ago) came from our success in running essentially all the 
examples through formal, mechanized proofs. Verification tools such as Boogie, 
while still a work in progress, have now reached a level of quality and practicality 
that enables them to provide resounding guarantees for work that, in the past, 
would have remained subject to human errors. 

Wc hope that others will continue this work, both on the conceptual side by 
providing further insights into the concept of loop invariant, extending it to its 
counterpart on the data side, the class invariant, and broadening our exploration 
and classification effort to many other important algorithms of computer science. 
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